{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d9afd03a",
   "metadata": {},
   "source": [
    "分布式锁、ELK stack、kafka、redis、分布式事务\n",
    "\n",
    "分布式：https://mp.weixin.qq.com/s/uyLLYkO8n8PYjZI4jAdg9w\n",
    "分布式事务：https://mp.weixin.qq.com/s/Da8m0mOKN9brMFUhekUzjA\n",
    "Redis:\n",
    "https://mp.weixin.qq.com/s/yh5qbWPjlGVyLcK1MwuOcA\n",
    "\n",
    "ES:https://mp.weixin.qq.com/s/3cTWVGZNFnmaOSFfiZZ_gg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1a686a7",
   "metadata": {},
   "source": [
    "- 中间件笔记\n",
    "    - 01-基础工具\n",
    "        - [1.1-Git核心用法与面试](中间件/01-基础工具/1.1-Git核心用法与面试.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bd8662a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "63c9d93d",
   "metadata": {},
   "source": [
    "提交代码前执行以下代码 去除ipynb 文件中的output cell"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6d56ccb",
   "metadata": {},
   "source": [
    "根据这个以下知识点和场景要求，给出所有对应知识点的面试理论讲解，再根据场景给出场景实战步骤"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "764feb4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import os\n",
    "import json\n",
    "import argparse\n",
    "\n",
    "def clear_ipynb_output(ipynb_path):\n",
    "    \"\"\"清除单个ipynb文件的输出内容\"\"\"\n",
    "    try:\n",
    "        with open(ipynb_path, 'r', encoding='utf-8') as f:\n",
    "            nb = json.load(f)\n",
    "        \n",
    "        # 遍历所有单元格，清除输出\n",
    "        for cell in nb.get('cells', []):\n",
    "            if 'outputs' in cell:\n",
    "                cell['outputs'] = []\n",
    "            if 'execution_count' in cell:\n",
    "                cell['execution_count'] = None\n",
    "        \n",
    "        # 保存清理后的文件\n",
    "        with open(ipynb_path, 'w', encoding='utf-8') as f:\n",
    "            json.dump(nb, f, ensure_ascii=False, indent=2)\n",
    "        \n",
    "        print(f\"已清理: {ipynb_path}\")\n",
    "        return True\n",
    "    except Exception as e:\n",
    "        print(f\"处理 {ipynb_path} 时出错: {str(e)}\")\n",
    "        return False\n",
    "\n",
    "def clear_all_ipynb_in_directory(directory, recursive=False):\n",
    "    \"\"\"清除目录下所有ipynb文件的输出内容\"\"\"\n",
    "    count = 0\n",
    "    for root, dirs, files in os.walk(directory):\n",
    "        for file in files:\n",
    "            if file.endswith('.ipynb'):\n",
    "                file_path = os.path.join(root, file)\n",
    "                if clear_ipynb_output(file_path):\n",
    "                    count += 1\n",
    "        # 如果不递归处理子目录，就只处理当前目录\n",
    "        if not recursive:\n",
    "            break\n",
    "    print(f\"总计清理了 {count} 个ipynb文件\")\n",
    "\n",
    "# if __name__ == \"__main__\":\n",
    "#     parser = argparse.ArgumentParser(description='清除目录下所有ipynb文件的输出内容')\n",
    "#     parser.add_argument('--dir', default='.', help='目标目录，默认为当前目录')\n",
    "#     parser.add_argument('--recursive', action='store_true', help='是否递归处理子目录')\n",
    "#     args = parser.parse_args()\n",
    "    \n",
    "# clear_all_ipynb_in_directory(args.dir, args.recursive)\n",
    "clear_all_ipynb_in_directory('.', True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed8b3112",
   "metadata": {},
   "source": [
    "Elastic Stack\n",
    "\n",
    "Elastic Stack 的可观测性由 Logs 、APM（application performance monitor）、Uptime、Metrics 四个模块组成，他们分别由四个组件提供支持。\n",
    "Logs -- Filebeat\n",
    " APM -- APM Server & APM agent\n",
    " Uptime -- Heartbeat\n",
    " Metrics -- Metricbeat\n",
    "\n",
    "Elastic Stack 是一系列由 Elastic 公司开发的产品组件，能够安全可靠地获取任何来源、任何格式的数据，然后实时地对数据进行搜索、分析和可视化。Elastic Stack旧称 ELK Stack，主要有 Elasticsearch，Logstash，Kibana，Beats 四种组件组成。\n",
    "\n",
    "由于一些原因，在 Elasticsearch 6.0 以后，一个 Index 只能含有一个 type。这其中的原因是：相同 index 的不同映射 type 中具有相同名称的字段是相同； 在Elasticsearch 索引中，不同映射 type 中具有相同名称的字段在 Lucene 中被同一个字段支持。在默认的情况下是 _doc。在未来 8.0 的版本中，type 将被彻底删除。\n",
    "\n",
    "一个 index 是一个逻辑命名空间，它映射到一个或多个主分片，并且可以具有零个或多个副本分片。shard 数目是不可以动态修改的，但是replica 的数目是可以动态修改的。 shard 数量不能改的原因：因为 Elasticsearch 会依据每个 document 的 id 及 primary shard 的数量来把相应的 document 分配到相应的 shard 中。如果这个数量以后修改的话，那么每次搜索的时候，可能会找不到相应的 shard。主分片和副本 之间的主要区别在于，只有主分片可以接受索引请求。副本和主分片都可以提供查询请求。\n",
    "\n",
    "分片：\n",
    "有两种类型的分片：Primary shard 和 Replica shard。\n",
    "\n",
    "shard 健康：\n",
    " 红色：集群中未分配至少一个主分片\n",
    " 黄色：已分配所有主副本，但未分配至少一个副本\n",
    " 绿色：分配所有分片\n",
    "\n",
    "master-eligible：可以作为主 node。一旦成为主 node，它可以管理整个 cluster的设置及变化：创建，更新，删除 index；添加或删除 node；为 node 分配 shard\n",
    " data：数据 node\n",
    " ingest: 数据接入（比如 pipepline)\n",
    " machine learning (Gold/Platinum License)\n",
    "\n",
    "Elasticsearch 具有 schema-less 的能力，这意味着无需显式指定如何处理文档中可能出现的每个不同字段，即可对文档建立索引。 启用动态映射后，Elasticsearch 自动检测并向索引添加新字段。 这种默认行为使索引和浏览数据变得容易-只需开始建立索引文档，Elasticsearch 就会检测布尔值，浮点数和整数值，日期和字符串，并将其映射到适当的 Elasticsearch 数据类型。\n",
    "\n",
    "\n",
    "ES 节点启动的默认需求为 1C2G （1 核 CPU，2GB 内存）\n",
    "一般的内存、CPU 配置策略大致为以下几种：\n",
    " master 节点需要适当大小的内存\n",
    " coordination 节点需要较大的内存和 CPU\n",
    " ingest 节点需要较大的 CPU\n",
    " data 节点需要较大的内存和硬盘\n",
    "\n",
    "\n",
    "没做特殊配置的话，ES 会在写入及不断的查询过程中，将数据集中存在最新修改和召回的数据缓存在节点 /集群的各级内存（缓存）中。同时将绝大部分数据存在磁盘中的各种索引文件中，仅在内存中保留一部分索引文件的索引以加速数据的读写。\n",
    " 不同于内存中的文件，ES 放置在磁盘中的文件的读写是随机的不是顺序的，所以更快的随机读写速度将使 ES 提供更快的数据存取速度。\n",
    " 在在线搜索等高频存取的场景中，更建议使用固态硬盘以支持数据的高速读写。 在离线的日志存储等低频读取的场景中，则可以考虑用机械硬盘来节约成本。\n",
    "\n",
    "Elasticsearch 使用\n",
    "一种称为倒排索引的结构，它适用于快速的全文搜索。一个倒排索引由文档中所有不重复词的列表构成，对于其中每个词，有一个包含它的文档列表。\n",
    "\n",
    "\n",
    "Doc Values 是在索引时创建的，当字段索引时，Elasticsearch 为了能够快速检索，会把字段的值加入倒排索引中，同时它也会存储该字段的 Doc Values。\n",
    "在 Elasticsearch 中，Doc Values 是一种列式存储结构，用于支持高效的排序、聚合、过滤和脚本操作。它与倒排索引（用于全文搜索）相辅相成，共同构成了 Elasticsearch 数据存储和查询的核心机制。\n",
    "Elasticsearch 中的 Doc Values 常被应用到以下场景：\n",
    " 对一个字段进行排序\n",
    " 对一个字段进行聚合\n",
    " 某些过滤，比如地理位置过滤\n",
    " 某些与字段相关的脚本计算\n",
    " 使用 docvalue_fields 返回搜索结果部分字段值\n",
    "\n",
    "\n",
    "默认情况下，字段原始值会被索引用于查询，但是不会被存储。为了展示文档内容，通过一个叫 _source 的字段用于存储整个文档的原始值。\n",
    "_source 字段包含索引时发送的原始 JSON 文档。_source 字段本身不建索引，但是存储原始文档，以便在执行查询请求时，可以将其返回。可以通过设置，禁用原文字段，或者只存储特定字段。\n",
    "_source 在 Lucene 中是映射为一个特殊的字段：\n",
    "Field 由Index、IndexType、Anayzer、DocValue、Store 来设置\n",
    "\n",
    "\n",
    "Elasticsearch 支持如下简单字段类型：\n",
    " 字符串： text，keyword\n",
    " 整数：byte，short，integer，long\n",
    " 浮点数： float，double\n",
    " 布尔型： boolean\n",
    " 日期： date\n",
    "\n",
    "尽管在很多情况下基本字段数据类型已经够用，但你经常需要为单独字段自定义映射，特别是字符串字段。自定义映射允许你执行下面的操作：\n",
    " 全文字符串字段和精确值字符串字段的区别\n",
    " 使用特定语言分析器\n",
    " 优化字段以适应部分匹配\n",
    " 指定自定义数据格式\n",
    " 还有更多\n",
    "text 类型字段的最重要属性是分析器 analyzer，默认 Elasticsearch 使用standard 分析器， 但你可以指定一个内置的分析器替代它，例如 whitespace 、simple 、english、cjk："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bed23dd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Welcome to the Dev Tools Console!\n",
    "#\n",
    "# You can use Console to explore the Elasticsearch API. See the Elasticsearch API reference to learn more:\n",
    "# https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html\n",
    "#\n",
    "# Here are a few examples to get you started.\n",
    "\n",
    "\n",
    "# Create an index\n",
    "PUT /my-index\n",
    "\n",
    "\n",
    "# Add a document to my-index\n",
    "POST /my-index/_doc\n",
    "{\n",
    "    \"id\": \"park_rocky-mountain\",\n",
    "    \"title\": \"Rocky Mountain\",\n",
    "    \"description\": \"Bisected north to south by the Continental Divide, this portion of the Rockies has ecosystems varying from over 150 riparian lakes to montane and subalpine forests to treeless alpine tundra.\"\n",
    "}\n",
    "\n",
    "\n",
    "# Perform a search in my-index\n",
    "GET /my-index/_search?q=\"rocky mountain\"\n",
    "\n",
    "# 获取index 健康状况\n",
    "GET /_cat/indices/my-index\n",
    "# green open my-index kGlGcQZ0SdO_aEm206uWPQ 1 1 1 0 14.6kb 7.3kb 7.3kb\n",
    "\n",
    "\n",
    "# doc_values: false 如果你知道你永远也不会对某些字段进行聚合、排序或是使用脚本操作，你可以通过禁用特定字段的 Doc Values。这样不仅节省磁盘空间，也会提升索引的速度。\n",
    "# index:false 禁用倒排索引\n",
    "# store:true 默认情况下，字段原始值会被索引用于查询，但是不会被存储。为了展示文档内容，通过一个叫 _source 的字段用于存储整个文档的原始值。\n",
    "PUT my_index\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"session_id\": {\n",
    "        \"type\": \"keyword\",\n",
    "        \"doc_values\": false\n",
    "      },\n",
    "      \"customer_token\": {\n",
    "        \"type\": \"keyword\",\n",
    "        \"doc_values\": true,\n",
    "        \"index\": false\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# \n",
    "GET /my-index/_search\n",
    "{\n",
    "  \"stored_fields\": [\n",
    "    \"id\",\n",
    "    \"title\"  ]\n",
    "}\n",
    "\n",
    "# 禁用_source、_source字段包含/排除部分字段\n",
    "PUT my-index-000001\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"_source\": {\n",
    "      \"includes\":[\n",
    "        \"*.count\",\n",
    "        \"meta.*\"\n",
    "      ],\n",
    "      \"excludes\":[\n",
    "        \"meta.description\",\n",
    "        \"meta.other.*\"\n",
    "      ],\n",
    "      \"enabled\": false\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "DELETE twitter\n",
    "PUT twitter\n",
    "{\n",
    "    \"settings\": {\n",
    "        \"number_of_shards\": \"5\",\n",
    "        \"number_of_replicas\": \"1\"\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"user\": {\n",
    "                \"type\": \"keyword\"\n",
    "            },\n",
    "            \"post_date\": {\n",
    "                \"type\": \"date\"\n",
    "            },\n",
    "            \"message\": {\n",
    "                \"type\": \"text\",\n",
    "                \"analyzer\": \"cjk\"\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "# 添加字段\n",
    "PUT twitter/_mapping\n",
    "{\n",
    "  \"properties\": {\n",
    "    \"tag\": {\n",
    "      \"type\": \"keyword\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "GET twitter/_mapping\n",
    "\n",
    "GET /twitter/_analyze\n",
    "{\n",
    "  \"field\":\"message\",\n",
    "  \"text\":\"搜索索引引擎\"\n",
    "}\n",
    "GET /twitter/_analyze\n",
    "{\n",
    "  \"field\":\"tag\",\n",
    "  \"text\":\"搜索引擎\"\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75b46bbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "Document API\n",
    "\n",
    "PUT my_good\n",
    "{\n",
    "  \"settings\": {\n",
    "    \"index\": {\n",
    "      \"number_of_shards\": 1,\n",
    "      \"number_of_replicas\": 1\n",
    "    }\n",
    "  },\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"goodsName\": {\n",
    "        \"type\": \"text\",\n",
    "        \"analyzer\": \"ik_smart\"\n",
    "      },\n",
    "      \"skuCode\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"brandName\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"channelType\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"shopCode\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"publicPrice\": {\n",
    "        \"type\": \"float\"\n",
    "      },\n",
    "      \"closeUserCode\": {\n",
    "        \"type\": \"text\",\n",
    "        \"analyzer\": \"standard\"\n",
    "      },\n",
    "      \"boostValue\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"groupPrice\": {\n",
    "        \"type\": \"nested\",\n",
    "        \"properties\": {\n",
    "          \"boxLevelPrice\": {\n",
    "            \"type\": \"float\"\n",
    "          },\n",
    "          \"level\": {\n",
    "            \"type\": \"text\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "# 以 POST //_create/<_id>为例，以下将创建文档 ID 为 1 的商品信息：\n",
    "POST /my_goods/_create/1\n",
    "{\n",
    "  \"goodsName\": \"苹果 51 英寸 4K 超高清\",\n",
    "  \"skuCode\": \"skuCode1\",\n",
    "  \"brandName\": \"苹果\",\n",
    "  \"closeUserCode\": [\n",
    "    \"0\"\n",
    "  ],\n",
    "  \"channelType\": \"cloudPlatform\",\n",
    "  \"shopCode\": \"sc00001\",\n",
    "  \"publicPrice\": \"8188.88\",\n",
    "  \"groupPrice\": null,\n",
    "  \"boxPrice\": null,\n",
    "  \"boostValue\": 1.8\n",
    "}\n",
    "\n",
    "# bulk 批量操作\n",
    "POST my_goods/_bulk\n",
    "{\"index\":{\"_id\":1}}\n",
    "{\"goodsName\":\"苹果 51英寸 4K超高清\",\"skuCode\":\"skuCode1\",\"brandName\":\"苹果\",\"closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8188.88\",\"groupPrice\":null,\"boxPrice\":null,\"boostValue\":1.8}\n",
    "{\"index\":{\"_id\":2}}\n",
    "{\"goodsName\":\"苹果 55英寸 3K超高清\",\"skuCode\":\"skuCode2\",\"brandName\":\"苹果\",\"closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00002\",\"publicPrice\":\"6188.88\",\"groupPrice\":null,\"boxPrice\":null,\"boostValue\":1.0}\n",
    "{\"index\":{\"_id\":3}}\n",
    "{\"goodsName\":\"苹果 UA55RU7520JXXZ 53 英寸 4K 高清\",\"skuCode\":\"skuCode3\",\"brandName\":\"美国苹果\",\"closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8388.88\",\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"htd003\",\"uc004\"],\"boxPriceDetail\":4388.88},{\"boxType\":\"box2\",\"boxUserCode\":[\"uc005\",\"uc0010\"],\"boxPriceDetail\":5388.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":4}}\n",
    "{\"goodsName\":\"山东苹果 UA55RU7520JXXZ 苹果 54 英寸 5K 超高清\",\"skuCode\":\"skuCode4\",\"brandName\":\"山东苹果\",\"closeUserCode\":[\"uc001\",\"uc002\",\"uc003\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8488.88\",\"groupPrice\":[{\"level\":\"level1\",\"boxLevelPrice\":\"2488.88\"},{\"level\":\"level2\",\"boxLevelPrice\":\"3488.88\"}],\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":\n",
    "[\"uc004\",\"uc005\",\"uc006\",\"uc001\"],\"boxPriceDetail\":4488.88},{\"boxType\":\"box2\",\"boxUserCode\":\n",
    "[\"htd007\",\"htd008\",\"htd009\",\"uc0010\"],\"boxPriceDetail\":5488.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":5}}\n",
    "{\"goodsName\":\"苹果 UA55R 苹果 U7 苹果 520JXXZ 55 英寸 5K 超高清\",\"skuCode\":\"skuCode5\",\"brandName\":\"三星苹果\",\"closeUserCode\":[\"uc001\",\"uc002\",\"uc003\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8488.88\",\"groupPrice\":[{\"level\":\"level1\",\"boxLevelPrice\":\"\n",
    "2500\"},{\"level\":\"level2\",\"boxLevelPrice\":\"3500\"}],\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"uc\n",
    "004\",\"uc005\",\"uc006\",\"uc001\"],\"boxPriceDetail\":3588.88},{\"boxType\":\"box2\",\"boxUserCode\":[\"htd007\",\"htd008\",\"htd009\",\"uc0010\"],\"boxPriceDetail\":5588.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":6}}\n",
    "{\"goodsName\":\"三星UA55RU7520JXXZ 51英寸 4K超高清\",\"skuCode\":\"skuCode1\",\"brandName\":\"三星\",\"closeUserCode\":[\"0\"],\"channelType\":\"cmccPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8188.88\",\"groupPrice\":null,\"boxPrice\":null,\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":7}}\n",
    "{\"goodsName\":\"三星UA55RU7520JXXZ 52英寸 4K超高清\",\"skuCode\":\"skuCode2\",\"brandName\":\"三星\",\"closeUserCode\":[\"0\"],\"channelType\":\"cmccPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8288.88\",\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"htd002\"],\"boxPriceDetail\":4288.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":8}}\n",
    "{\"goodsName\":\"三星UA55RU7520JXXZ 52英寸 4K超高清\",\"skuCode\":\"skuCode2\",\"brandName\":\"三星\",\"closeUserCode\":[\"uc0022\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8288.88\",\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"uc0022\"],\"boxPriceDetail\":4288.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":9}}\n",
    "{\"goodsName\":\"三星UA55RU7520JXXZ 52英寸 4K超高清\",\"skuCode\":\"skuCode2\",\"brandName\":\"三星\",\"closeUserCode\":[\"uc0022\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8288.88\",\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"uc0022\"],\"boxPriceDetail\":4288.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":10}}\n",
    "{\"goodsName\":\"三星UA55RU7520JXXZ 52英寸 4K超高清\",\"skuCode\":\"skuCode2\",\"brandName\":\"三星\",\"closeUserCode\":[\"uc0022\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8288.88\",\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"uc0022\"],\"boxPriceDetail\":4288.88}],\"boostValue\":1.8}\n",
    "\n",
    "# DELETE /<index>/_doc/<_id>  DELETE /my_goods/_doc/2  _delete_by_query\n",
    "POST /my_goods/_delete_by_query\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"shopCode\": \"sc00002\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 更新 POST /<index>/_update/<_id>\n",
    "# 新增 字段\n",
    "POST /my_goods/_update/1\n",
    "{\n",
    "  \"doc\": {\n",
    "    \"shopName\": \"小王店铺\"\n",
    "  }\n",
    "}\n",
    "# 修改字段值\n",
    "POST /my_goods/_update/1\n",
    "{\n",
    "  \"doc\": {\n",
    "    \"shopName\": \"张三店铺\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 使用PUT 进行修改，需要罗列所有字段\n",
    "\n",
    "# 使用脚本实现更新  开启配置apack.tenant.allowed_scripts\n",
    "POST my_goods/_update/10\n",
    "{\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.city=params.channelType\",\n",
    "    \"lang\": \"painless\",\n",
    "    \"params\": {\n",
    "      \"channelType\": \"cloudPlatform1\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "GET /my_goods/_doc/2\n",
    "\n",
    "# 使用脚本 查询后更新 \n",
    "POST /my_goods/_update_by_query\n",
    "{\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.publicPrice=5888.00\",\n",
    "    \"lang\": \"painless\"\n",
    "  },\n",
    "  \"query\": {\n",
    "    \"term\": {\n",
    "      \"shopCode\": \"sc00002\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "#重建索引 reindex \n",
    "# 将 skuCode=skuCode2 的商品信息 reindex 到索引 my_goods_new\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my_goods\",\n",
    "    \"query\": {\n",
    "      \"match\": {\n",
    "        \"skuCode\": \"skuCode2\"\n",
    "      }\n",
    "    }\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my_goods_new\"\n",
    "  }\n",
    "}\n",
    "GET /my_goods_new/_search\n",
    "\n",
    "# GET <index>/_doc/<_id>\n",
    "# HEAD <index>/_doc/<_id>\n",
    "# GET <index>/_source/<_id>\n",
    "# HEAD <index>/_source/<_id>\n",
    "# 查询文档ID 1\n",
    "GET /my_goods/_doc/1\n",
    "\n",
    "#判断文档是否存在，返回信息更少，性能更高,\n",
    "# 存在：返回 200 - OK\n",
    "# 不存在：返回 404 - Not Found\n",
    "HEAD /my_goods/_doc/1\n",
    "\n",
    "# 查询只返回source\n",
    "GET /my_goods/_source/1\n",
    "# 查询返回 source 部分字段\n",
    "GET my_goods/_source/1/?_source_includes=brandName,goodsName\n",
    "\n",
    "#POST body 请求模式\n",
    "POST my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match_all\": {}\n",
    "  },\n",
    "  \"fields\": [\n",
    "    \"brandName\",\n",
    "    \"goodsName\"\n",
    "  ],\n",
    "  \"_source\": false\n",
    "}\n",
    "\n",
    "# Mutil get 批量查询\n",
    "GET /my_goods/_mget\n",
    "{   \n",
    "  \"docs\": [\n",
    "    {\n",
    "      \"_id\": \"1\"\n",
    "    },\n",
    "    {\n",
    "      \"_id\": \"2\"\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1c28160",
   "metadata": {},
   "source": [
    "Query DSL\n",
    "\n",
    "通常 Search 与 Filter 区别\n",
    "二者的查询是有区别的：\n",
    " Query 查询\n",
    "用于解答文档是否存在，并且告知返回文档与查询条件的匹配度，返回 _score 评分供用户选择。\n",
    " Filter 查询\n",
    "只用于返回文档是否与查询匹配，但是不会告诉你匹配度，即不进行评分。在做聚合查询时，filter 经常发挥更大的作用。因为没有评分 Elasticsearch 的处理速度就会提高，提升了整体响应时间。同时 filter 可以缓存查询结果，而 Query 则不能缓存。\n",
    "使用场景\n",
    "如果涉及到全文检索以及评分相关业务使用 Query，其他场景推荐使用 Filter 查询。\n",
    "\n",
    "Boolean 查询包含 must、filter、must_not。\n",
    "must ：必须匹配并且返回评分，filter 忽略评分，should 相当于数据库查询中的or，针对 should 有一个特殊的情况，也就是所有的搜索只有 should ，那么必须满足should 里的其中一个才会被搜索到。must_not 为不匹配，相当于不等于。\n",
    "minimum_should_match 为最小匹配数量，如果 bool 查询包含至少一个 should子句，并且没有 must 或 filter 子句，则默认值为 1，否则，默认值为 0。\n",
    "\n",
    "Booting查询：Boosting 用于控制评分相关度相关，可以提升评分也可以降低评分。boosting 查询 negative_boost 在0到1之间用于降低评分，相反，大于1 用于提升评分 （我不懂）\n",
    "在 Elasticsearch 中，negative_boost 是一个特殊的查询参数，用于降低匹配特定条件的文档的评分，与 boost（提升评分）的作用相反。它的核心功能是：让符合某些 “不希望优先出现” 条件的文档在搜索结果中排得更靠后，从而间接提升其他文档的相对优先级。\n",
    "\n",
    "如果需要同时处理多个正向和负向条件，可以在 bool 查询中嵌套 boosting，或直接用 should 子句配合 negative_boost。\n",
    "1、优先展示 “自营” 商品（boost 提升）； 2、其次展示 “第三方正品”；3、最后展示 “二手” 商品（negative_boost 压低）。\n",
    "{\n",
    "  \"query\": {\n",
    "    \"bool\": {\n",
    "      \"must\": [\n",
    "        {\"match\": {\"name\": \"笔记本电脑\"}}  // 必须包含的基础条件\n",
    "      ],\n",
    "      \"should\": [\n",
    "        // 正向条件：自营商品权重提升\n",
    "        {\"term\": {\"seller_type\": {\"value\": \"自营\", \"boost\": 2.0}}},\n",
    "        // 负向条件：二手商品权重降低（用 boosting 嵌套）\n",
    "        {\n",
    "          \"boosting\": {\n",
    "            \"positive\": {\"match_all\": {}},  // 匹配所有文档（仅作为容器）\n",
    "            \"negative\": {\"term\": {\"condition\": \"二手\"}},\n",
    "            \"negative_boost\": 0.4  // 二手商品评分乘以 0.4\n",
    "          }\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "Constant score query 查询：当查询不关心 TF（词频）时，就可以使用 constant score query\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c30a45ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "GET /my_goods/_mget\n",
    "{   \n",
    "  \"docs\": [\n",
    "    {\n",
    "      \"_id\": \"1\"\n",
    "    },\n",
    "    {\n",
    "      \"_id\": \"2\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\n",
    "# 如果 bool 查询包含至少一个 should子句，并且没有 must 或 filter 子句，则默认值为 1，否则，默认值为 0。\n",
    "POST /my_goods/_search\n",
    "{\n",
    "    \"query\": {\n",
    "        \"bool\": {\n",
    "            \"must\": {\n",
    "                \"term\": {\n",
    "                    \"shopCode\": \"sc00001\"\n",
    "                }\n",
    "            },\n",
    "            \"filter\": {\n",
    "                \"term\": {\n",
    "                    \"channelType\": \"cloudPlatform\"\n",
    "                }\n",
    "            },\n",
    "            \"must_not\": [\n",
    "                {\n",
    "                    \"range\": {\n",
    "                        \"publicPrice\": {\n",
    "                            \"gte\": 8288,\n",
    "                            \"lte\": 8888\n",
    "                        }\n",
    "                    }\n",
    "                }\n",
    "            ],\n",
    "            \"should\": [\n",
    "                {\n",
    "                    \"term\": {\n",
    "                        \"brandName\": {\n",
    "                            \"value\": \"果\"\n",
    "                        }\n",
    "                    }\n",
    "                }\n",
    "            ],\n",
    "            \"minimum_should_match\": 1\n",
    "        }\n",
    "    }\n",
    "}\n",
    "# minimum_should_match \n",
    "# minimum_should_match 为最小匹配数量，如果 bool 查询包含至少一个 should子句，并且没有 must 或 filter 子句，则默认值为 1，否则，默认值为 0。\n",
    "POST /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"bool\": {\n",
    "      \"should\": [\n",
    "        {\n",
    "          \"term\": {\n",
    "            \"brandName\": {\n",
    "              \"value\": \"东\"\n",
    "            }\n",
    "          }\n",
    "        },\n",
    "        {\n",
    "          \"term\": {\n",
    "            \"brandName\": {\n",
    "              \"value\": \"果\"\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "      ],\n",
    "      \"minimum_should_match\": 1\n",
    "    }\n",
    "  },\n",
    "  \"_source\":true\n",
    "}\n",
    "GET /my_goods/_doc/3\n",
    "#boosting 查询 negative_boost 在0到1之间用于降低评分，相反，大于1 用于提升评分 （我不懂）\n",
    "POST /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"boosting\": {\n",
    "      \"positive\": {\n",
    "        \"match\": {\n",
    "          \"skuCode\": \"skuCode1\"\n",
    "        }\n",
    "      },\n",
    "      \"negative\": {\n",
    "        \"match\": {\n",
    "          \"goodsName\": \"三星\"\n",
    "        }\n",
    "      },\n",
    "      \"negative_boost\": 0.2\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#Constant score query 查询：当查询不关心 TF（词频）时，就可以使用 constant score query\n",
    "POST /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"constant_score\": {\n",
    "      \"filter\": {\n",
    "        \"match\": {\n",
    "          \"goodsName\": \"苹果\"\n",
    "        }\n",
    "      },\n",
    "      \"boost\": 1.2\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Disjunction max query 查询\n",
    "# Disjunction 查询也被理解为分离最大化查询，指的是将任何与任一查询匹配的文档，作为结果返回，但只将最佳匹配的评分，作为查询的评分结果返回。\n",
    "# 例如查询商品名称和品牌名称中包含“苹果”的信息，当品牌的评分高于商品名称时，则返回品牌的评分做为总评分（忽略 tie_breaker 缓冲）。\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"dis_max\": {\n",
    "      \"tie_breaker\": 0.7,\n",
    "      \"boost\": 1.2,\n",
    "      \"queries\": [\n",
    "        {\n",
    "          \"match\": {\n",
    "            \"goodsName\": \"苹果\"\n",
    "          }\n",
    "        },\n",
    "        {\n",
    "          \"match\": {\n",
    "            \"brandName\": \"苹果\"\n",
    "          }\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Function score query 查询\n",
    "# Function score 允许你控制查询评分，是用来控制评分过程的终极武器。最高效的用法是用过滤器对结果的子集应用不同的函数，同时运用了 filter 的缓存，并且达到控制评分的过程。\n",
    "# score_mode\n",
    "# multiply: 默认，分数相乘\n",
    "# avg：平均分数，第一个 function 的分数\n",
    "# max：使用评分最大的分数\n",
    "# min：使用评分最小的分数 avg\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"function_score\": {\n",
    "      \"query\": {\n",
    "        \"match\": {\n",
    "          \"goodsName\": \"苹果\"\n",
    "          \n",
    "        }\n",
    "      },\n",
    "      \"boost\": 2,\n",
    "      \"functions\": [\n",
    "        {\n",
    "          \"filter\": {\n",
    "            \"match\": {\n",
    "              \"brandName\": \"美国\"\n",
    "            }\n",
    "          },\n",
    "          \"random_score\": {},\n",
    "          \"weight\": 2\n",
    "        },\n",
    "        {\n",
    "          \"filter\": {\n",
    "            \"match\": {\n",
    "              \"brandName\": \"山东\"\n",
    "            }\n",
    "          },\n",
    "          \"weight\": 40\n",
    "        }\n",
    "      ],\n",
    "      \"max_boost\": 60,\n",
    "      \"score_mode\": \"max\",\n",
    "      \"boost_mode\": \"multiply\",\n",
    "      \"min_score\": 2\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35a1944e",
   "metadata": {},
   "source": [
    "Full Text 查询\n",
    "\n",
    "| Match | 返回匹配查询条件的文档内容，查询条件会在匹配之前会被分词处理。\n",
    "\n",
    "match_phrase\n",
    "将查询条件的中的信息看做一个整体，如下面的 “goods t” 必须 goods 在前 t在后。在 match_phrase 中，可以通过 slop 来控制单词中间的间隔，默认为 0，\n",
    "match\n",
    "将查询中的条件做分词处理后，再去做查询。\n",
    "\n",
    "\n",
    "match 四种查询\n",
    "Match boolean prefix：是一个 Boolean 查询，将分词后的短语按照 term 进行查询，最后一个词组按照 prefix 查询。\n",
    "match phrase:| 将查询条件当做一个词组进行查询，不进行分词处理。\n",
    "Match phrase prefix : 返回文档包含给定查询条件的文档，文档中必须包含给定条件的内容且是按照顺序的，与 match phrase 类似，对最后一个 token 会进行前缀匹配，可以通过 slop来控制匹配 token 的位置差。\n",
    "Multi-match:多字段匹配，可以在多个字段中匹配查询相关信息，通过 type 参数可以调整结果集：\n",
    "    best_fields ：默认，匹配 fields，将评分最高的分数做为整个查询的分数返回； most_fields：查询匹配的文档，并且返回各个字段的分数之和的平均值；\n",
    "     cross_fields：跨字段匹配，匹配多个字段中是否包含查询词组，对每个字段分别进行打分，然后执行 max 运算获取打分最高的；\n",
    "     phrase：以 match_phrase 方式运行查询，并返回最佳匹配的评分做为总评分； phrase_prefix：以 match_phrase_prefix 方式运行查询，并返回最佳匹配的评分做为总评分；\n",
    "     bool_prefix：在每个字段上运行 match_bool_prefix 查询，并组合每个字段的评分，详情参考 bool_prefix 以 cross_fields 为例进行实战讲解。\n",
    "\n",
    "\n",
    "Term -level:可以使用 Term - level 查询结构化数据，结构化数据如日期范围、IP 地址、价格等,下\n",
    "\n",
    "326"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e75e4095",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Full Text 查询\n",
    "# Match 查询 带有高亮效果\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"goodsName\": \"苹果 高清 英寸\"\n",
    "    }\n",
    "  },\n",
    "  \"highlight\": {\n",
    "    \"fields\": {\n",
    "      \"goodsName\": {\n",
    "        \"pre_tags\": [\n",
    "          \"<strong>\"\n",
    "        ],\n",
    "        \"post_tags\": [\n",
    "          \"</strong>\"\n",
    "        ]\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Match 查询 operator 默认 or \n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"goodsName\": {\n",
    "        \"query\":\"苹果 高清 英寸 U7\",\n",
    "        \"operator\":\"and\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Match phrase query\n",
    "# 用于匹配索引中是否存在所输入的查询条件数据: slop 控制单词中间的间隔，默认为0\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match_phrase\": {\n",
    "      \"goodsName\":\n",
    "      {\n",
    "        \"query\": \"山东 5K\",\n",
    "        \"slop\": 8\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "POST my_goods/_bulk\n",
    "{\"index\":{\"_id\":13}}\n",
    "{\"goodsName\":\"apple and goods product \",\"skuCode\":\"skuCode3\",\"brandName\":\"美国苹果\",\"\n",
    "closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8388.88\",\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"htd003\",\"uc004\"],\"boxPriceDetail\":4388.88},{\"boxType\":\"box2\",\"boxUserCode\":[\"uc005\",\"uc0010\"],\"boxPriceDetail\":5388.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":21}}\n",
    "{\"goodsName\":\"apple goods test\",\"skuCode\":\"skuCode3\",\"brandName\":\"美国苹果\",\"closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8388.88\",\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"htd003\",\"uc004\"],\"boxPriceDetail\":4388.88},{\"boxType\":\"box2\",\"boxUserCode\":[\"uc005\",\"uc0010\"],\"boxPriceDetail\":5388.88}],\"boostValue\":1.2}\n",
    "\n",
    "# Match phrase prefix query\n",
    "# 返回文档包含给定查询条件的文档，文档中必须包含给定条件的内容，且是按照prefix 来进行匹配的，如 \"apple goods test\" ，商品名称包含 apple goods test 的数据将被查询到返回。\n",
    "\n",
    "#只返回 goodsName : apple goods test 的数据 匹配条件 goods test、apple、goods、\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match_phrase_prefix\": {\n",
    "      \"goodsName\": \"apple\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# multi_match 多字段匹配，可以在多个字段中匹配查询相关信息，通过 type 参数可以调整结果集：\n",
    "POST /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"multi_match\": {\n",
    "      \"query\": \"苹果\",\n",
    "      \"type\": \"best_fields\",\n",
    "      \"fields\": [\n",
    "        \"goodsName\",\n",
    "        \"brandName\"\n",
    "      ],\n",
    "      \"tie_breaker\": 0.3\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "PUT my_shop\n",
    "{\n",
    "  \"settings\": {\n",
    "    \"number_of_shards\": 1,\n",
    "    \"number_of_replicas\": 1\n",
    "  },\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"firstName\": {\n",
    "        \"type\": \"text\"\n",
    "      },\n",
    "      \"lastName\": {\n",
    "        \"type\": \"text\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "POST my_shop/_bulk\n",
    "{\"index\":{\"_id\":1}}\n",
    "{\"first_name\":\"Will\",\"last_name\":\"Smith\",\"age\":25}\n",
    "{\"index\":{\"_id\":2}}\n",
    "{\"first_name\":\"Smith\",\"last_name\":\"hello\",\"age\":21}\n",
    "{\"index\":{\"_id\":3}}\n",
    "{\"first_name\":\"Will\",\"last_name\":\"hello\",\"age\":20}\n",
    "\n",
    "# multi_match \n",
    "GET /my_shop/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"multi_match\": {\n",
    "      \"query\": \"Will Smith\",\n",
    "      \"type\": \"cross_fields\",\n",
    "      \"fields\": [\n",
    "        \"first_name^2\",\n",
    "        \"last_name\"\n",
    "      ],\n",
    "      \"operator\": \"and\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "# Term -Level\n",
    "# 返回包含 goodsName 字段的索引文档 Exists \n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"exists\": {\n",
    "      \"field\": \"goodsName\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Fuzzy 查询\n",
    "# 返回包含与搜索字词相似的字词的文档，可以用于查询纠错功能。\n",
    "# Edit distance 指的是最小编辑距离，指的是两个字符串之间，由一个字符串转换为\n",
    "# 另外一个字符串，所需要的最少编辑次数，也叫：Levenshtein ，\n",
    "#一些查询和 APIs 支持参数去做不精准查询操作，此时可以使用 fuzziness 参数\n",
    "# 0、1、2 表示最大允许可编辑距离\n",
    "#AUTO 根据词项的长度确定可编辑距离数值，有两种可选参数，AUTO:[low] 和[high]，用于分别表示短距离参数与长距离参数，未指定情况下，默认值是 3 和 6。\n",
    "#   0..2 单词长度为 0 到 2 个字母之间时，必须要精确匹配\n",
    "#   3..5 单词长度 3 到 5 个字母时，最大编辑距离为 1\n",
    "#   5 单词长度大于 5 个字母时，最大编辑距离为 2\n",
    "\n",
    "#以官网例子举例说明\n",
    "POST /my_index/_bulk\n",
    "{ \"index\": { \"_id\": 1 }}\n",
    "{ \"text\": \"Surprise me!\"}\n",
    "{ \"index\": { \"_id\": 2 }}\n",
    "{ \"text\": \"That was surprising.\"}\n",
    "{ \"index\": { \"_id\": 3 }}\n",
    "{ \"text\": \"I wasn't surprised.\"}\n",
    "\n",
    "GET /my_index/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"fuzzy\": {\n",
    "      \"text\": {\n",
    "        \"value\": \"surprize\",\n",
    "        \"prefix_length\": 1\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "GET /my_index/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"fuzzy\": {\n",
    "      \"text\": {\n",
    "        \"value\": \"surprize\",\n",
    "        \"fuzziness\": \"1\",\n",
    "        \"prefix_length\": 1\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Ids 查询\n",
    "# 范围文档包含 ID 的文档信息：\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"ids\": {\n",
    "      \"values\": [\n",
    "        \"1\",\n",
    "        \"4\",\n",
    "        \"5\"\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Prefix 查询\n",
    "# 返回在提供的字段中包含特定前缀的文档：\n",
    "PUT my_shop_test\n",
    "{\n",
    "  \"settings\": {\n",
    "    \"number_of_shards\": 1,\n",
    "    \"number_of_replicas\": 1\n",
    "  },\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"shopName\": {\n",
    "        \"type\": \"text\"\n",
    "      },\n",
    "      \"shopCode\": {\n",
    "        \"type\": \"text\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#添加测试数据\n",
    "POST my_shop_test/_bulk\n",
    "{\"index\":{\"_id\":1}}\n",
    "{\"shopName\":\"box\",\"shopCode\":\"Smith\"}\n",
    "{\"index\":{\"_id\":2}}\n",
    "{\"shopName\":\"black\",\"shopCode\":\"jack\"}\n",
    "{\"index\":{\"_id\":3}}\n",
    "{\"shopName\":\"fox\",\"shopCode\":\"act\"}\n",
    "{\"index\":{\"_id\":4}}\n",
    "{\"shopName\":\"booex\",\"shopCode\":\"act\"}\n",
    "\n",
    "GET /my_shop_test/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"prefix\": {\n",
    "      \"shopName\": {\n",
    "        \"value\": \"bo\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Range 查询\n",
    "# Range 查询类似数据库中的 大于、小于范围查询：\n",
    "GET my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"range\": {\n",
    "      \"publicPrice\": {\n",
    "        \"gte\": 2000,\n",
    "        \"lte\": 8488\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Regexp 查询\n",
    "# 正则表达式查询，查询店铺编码以 's' 开头，中间包括任何字符，以及长度且以'1'结尾的数据：\n",
    "GET my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"regexp\": {\n",
    "      \"shopCode\": {\n",
    "        \"value\": \"s.*1\",\n",
    "        \"flags\": \"ALL\",\n",
    "        \"case_insensitive\": true,\n",
    "        \"max_determinized_states\": 10000,\n",
    "        \"rewrite\": \"constant_score\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Term 查询\n",
    "# #返回确切的文档内容，避免对 text 字段类型使用 term\n",
    "GET my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"term\": {\n",
    "      \"brandName\": {\n",
    "        \"value\": \"三星\",\n",
    "        \"boost\": 1\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Terms 查询\n",
    "#.Terms 返回一个或多个包含精确查询条件的文档信息：\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"terms\": {\n",
    "      \"brandName\": [\n",
    "        \"美国\",\n",
    "        \"三星\"\n",
    "      ],\n",
    "      \"boost\": 1\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Terms_set 查询\n",
    "# 返回最小精确匹配成功的文档信息，terms_set 类似 terms 查询，只不过 terms_set 多定义了返回最小匹配的数量。\n",
    "#新定义商品信息\n",
    "PUT /my_goods_info\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"goodsName\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"sale_property\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"required_matches\": {\n",
    "        \"type\": \"long\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#添加 3 条商品测试数据\n",
    "#销售属性 白色、64G、标品\n",
    "PUT /my_goods_info/_doc/1?refresh\n",
    "{\n",
    "  \"name\": \"apple\",\n",
    "  \"sale_property\": [\n",
    "    \"white\",\n",
    "    \"64\",\n",
    "    \"standard\"\n",
    "  ],\n",
    "  \"required_matches\": 2\n",
    "}\n",
    "\n",
    "#黑色、32G、非标品\n",
    "PUT /my_goods_info/_doc/2?refresh\n",
    "{\n",
    "  \"name\": \"apple\",\n",
    "  \"sale_property\": [\n",
    "    \"black\",\n",
    "    \"32\",\n",
    "    \"no standard\"\n",
    "  ],\n",
    "  \"required_matches\": 2\n",
    "}\n",
    "\n",
    "#黑色、64 非标品\n",
    "PUT /my_goods_info/_doc/3?refresh\n",
    "{\n",
    "  \"name\": \"apple\",\n",
    "  \"sale_property\": [\n",
    "    \"black\",\n",
    "    \"64\",\n",
    "    \"no standard\"\n",
    "  ],\n",
    "  \"required_matches\": 2\n",
    "}\n",
    "\n",
    "#查询\n",
    "GET /my_goods_info/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"terms_set\": {\n",
    "      \"sale_property\": {\n",
    "        \"terms\": [\n",
    "          \"white\",\n",
    "          \"64\"\n",
    "        ],\n",
    "        \"minimum_should_match_field\": \"required_matches\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Wildcard 查询\n",
    "# 返回包含与通配符模式匹配的术语的文档：\n",
    "GET /my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"wildcard\": {\n",
    "      \"shopCode\": {\n",
    "        \"value\": \"sc*1\",\n",
    "        \"boost\": 1,\n",
    "        \"rewrite\": \"constant_score\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Geo 查询\n",
    "# Elasticsearch 支持两种 geo 数据：geo_point 经纬度 和 geo_shape 点、线、圆、多边形等复杂图形\n",
    "# Geo_point\n",
    "#用于查找距离另一个坐标范围内的所有坐标点，或者计算两点之间的距离用于排序、打分、聚合等操作。\n",
    "# Geo-shapes\n",
    "#常用于过滤，比如判断两个地理形状是否有重叠或者某个地形是否包含了其他的地\n",
    "#理形状\n",
    "\n",
    "#查询分为 4 种类型：\n",
    "# geo_bounding_box：查找具有落入指定矩形的地理位置的坐标点\n",
    "# geo_distance：查找地理点在中心点指定距离内的坐标点\n",
    "# geo_polygon：查找具有指定多边形内的地理点的坐标点 \n",
    "#  geo_shape：查找具有以下内容的坐标点：\n",
    "#      geo-shapes 与指定的几何形状相交，包含于其中或不与指定的几何形状相交的坐标点\n",
    "#      geo-points 与指定的地理形状相交的坐标点\n",
    "\n",
    "# Geo-bounding box 查询\n",
    "# 定义索引对象店铺信息：\n",
    "PUT /my_shop_info\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"pin\": {\n",
    "        \"properties\": {\n",
    "          \"location\": {\n",
    "            \"type\": \"geo_point\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#添加 2 条测试数据\n",
    "PUT /my_shop_info/_doc/1\n",
    "{\n",
    "  \"pin\": {\n",
    "    \"location\": {\n",
    "      \"lat\": 40.12,\n",
    "      \"lon\": -71.34\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "PUT /my_shop_info/_doc/2\n",
    "{\n",
    "  \"pin\": {\n",
    "    \"location\": {\n",
    "      \"lat\": 50.12,\n",
    "      \"lon\": -61.34\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#查询指定范围内的数据\n",
    "GET my_shop_info/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"bool\": {\n",
    "      \"must\": {\n",
    "        \"match_all\": {}\n",
    "      },\n",
    "      \"filter\": {\n",
    "        \"geo_bounding_box\": {\n",
    "          \"pin.location\": {\n",
    "            \"top_left\": {\n",
    "              \"lat\": 40.73,\n",
    "              \"lon\": -74.1\n",
    "            },\n",
    "            \"bottom_right\": {\n",
    "              \"lat\": 40.01,\n",
    "              \"lon\": -71.12\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "# Geo-distance 查询\n",
    "# 查询仅包含距某个地理点特定距离之内的匹配的坐标，如下所示，查询坐标：\n",
    "#仍然以 my_shop_info 为例\n",
    "GET /my_shop_info/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"bool\": {\n",
    "      \"must\": {\n",
    "        \"match_all\": {}\n",
    "      },\n",
    "      \"filter\": {\n",
    "        \"geo_distance\": {\n",
    "          \"distance\": \"200km\",\n",
    "          \"pin.location\": {\n",
    "            \"lat\": 40,\n",
    "            \"lon\": -70\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fb1147a",
   "metadata": {},
   "source": [
    "单机Lucene打分机制\n",
    "\n",
    "其中 q 表示查询，d 表示当前文档，t 表示 q 中的词条，tf(t in d) 是计算词条t 在文档 d 中的词频，idf(t) 是词条 t 在整个索引中的逆文档频率。\n",
    "我们介绍一下最关键的两个概念，即词频（ TF ）和逆文档频率( IDF )。\n",
    "词频（ TF ）：词条在文档中出现的次数\n",
    "基于特定的 q 和文档 d 来说，词条 t 代指 q 分词后的其中一个词条，t 的词频指该 t 词条在文档 d 中的出现次数，出现次数越多，表示该文档相对于该词关联度更高。\n",
    "\n",
    "逆文档频率（IDF ）：在同一索引中存在该词条的文档数的倒数\n",
    "包含某个词条的文档数越多，说明这个词条的词频在整个索引中的影响力越弱。\n",
    "\n",
    "对于该公式的其他各项的含义，本小节不作深入介绍，我们仅需了解，一旦给定查询 q 和文档 d，其得分即为查询中每个词条 t 的得分总和。\n",
    "而每个词条的得分，一个主要部分是该词条在文档 d 中的词频 ( TF ) 乘以逆文档频率 ( IDF ) 的平方。即词条在文档中出现的频率越高，则得分越高。而索引中存在该词条的文档越少，逆文档频率则越高，表示该词条越罕见，那么对应的分数也将越高。\n",
    "\n",
    "\n",
    "Elasticsearch 打分机制\n",
    "Elasticsearch 的搜索类型有两种，默认的称为 QUERY_THEN_FETCH。顾名思义，它的搜索流程分为两个阶段，分别称之为 Query 和 Fetch 。\n",
    "我们来看下 QUERY_THEN_FETCH 的流程：\n",
    "Query 阶段：\n",
    " Elasticsearch 在收到客户端搜索请求后，会由协调节点将请求分发到对应索引的每个 Shard 上。\n",
    " 每个 Shard 的 Lucene 实例基于本地 Shard 内的 TF/IDF 统计信息，独立完成Shard 内的索引匹配和打分（基于上述公式），并根据打分结果完成单个 Shard内的排序、分页。\n",
    " 每个 Shard 将排序分页后的结果集的元数据（文档 ID 和分数，不包含具体的文档内容）返回给协调节点。\n",
    " 协调节点完成整体的汇总、排序以及分页，筛选出最终确认返回的搜索结果。\n",
    "Fetch 阶段：\n",
    " 协调节点根据筛选结果去对应 shard 拉取完整的文档数据\n",
    " 整合最终的结果返回给用户客户端\n",
    "\n",
    "通过分析上面的 QUERY_THEN_FETCH 流程，我们不难发现：由于分布式系统天然的割裂性质，每个 shard 无法看到全局的统计信息，所以上述第 2 步中每个 Shard的打分都是基于本地 Shard 内的 TF/IDF 统计信息来完成的。\n",
    "\n",
    "在大多数的生产环境中，由于数据量多且在每个 Shard 分布均匀，这种方式是没有问题的。但是在极端情况下（如上例），3 个 shard 中的文档数相差较大，那么IDF 在 3 个 Shard 中所起到的影响将截然不同，即单个 Shard 内打分汇总后的结果，与全局打分汇总的结果会有相当大的出入，造成我们在靠前的分页，搜到原本应该排名靠后的文档。\n",
    "这也是分布式打分引入的实际问题，那么如何才能解决这类问题呢？\n",
    "我们曾在上一小节提到，Elasticsearch 的搜索类型其实有两种，除了上面介绍的QUERY_THEN_FETCH 之外，还有一种是 DFS_QUERY_THEN_FETCH。\n",
    "DFS 在这里的意思是分布式频率打分，其思想是提前向所有 Shard 进行全局的统计信息搜集，然后再将这些统计信息，随着查询分发到各个 Shard，让各个 Shard 在本地采用全局 TF/IDF 来打分，具体的流程如下：\n",
    "预统计阶段：\n",
    " Elasticsearch 在收到客户端搜索请求后，会由协调节点进行一次预统计工作，即先\n",
    "向所有相关 Shard 搜集统计信息\n",
    "\n",
    "Query 阶段：\n",
    " 由协调节点整合所有统计信息，将全局的统计信息连同请求一起分发到对应索引的每个 Shard 上。\n",
    " 每个 Shard 的 Lucene 实例，基于全局的 TF/IDF 统计信息，独立完成 Shard 内的索引匹配和打分（基于上述公式），并根据打分结果，完成单个 Shard 内的排序、分页。\n",
    " 每个 Shard 将排序分页后的结果集的元数据（文档 ID 和分数，不包含具体的文档内容）返回给协调节点。\n",
    " 协调节点完成整体的汇总、排序以及分页，筛选出最终确认返回的搜索结果。\n",
    "Fetch 阶段：\n",
    " 协调节点根据筛选结果去对应 shard 拉取完整的文档数据\n",
    " 整合最终的结果返回给用户客户端\n",
    "综上可见，Elasticsearch 在分布式打分上做了权衡，如果要考虑绝对的精确性，那么需要牺牲一些性能来换取全局的统计信息。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7502024c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 分布式计分\n",
    "\n",
    "#删除已有索引\n",
    "DELETE /my-index-000001\n",
    "#创建索引，显示在 settings 中指定 2 个 shard:\"number_of_shards\": \"2\"\n",
    "PUT /my-index-000001\n",
    "{\n",
    "  \"settings\": {\n",
    "    \"number_of_shards\": \"2\",\n",
    "    \"number_of_replicas\": \"1\"\n",
    "  },\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"title\": {\n",
    "        \"type\": \"text\"\n",
    "      },\n",
    "      \"date\": {\n",
    "        \"type\": \"date\"\n",
    "      },\n",
    "      \"content\": {\n",
    "        \"type\": \"text\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#插入记录\n",
    "PUT /my-index-000001/_doc/1\n",
    "{\n",
    "  \"title\": \"三国志\",\n",
    "  \"date\": \"2021-05-01\",\n",
    "  \"content\": \"国别体史书\"\n",
    "}\n",
    "\n",
    "PUT /my-index-000001/_doc/2\n",
    "{\n",
    "  \"title\": \"红楼梦\",\n",
    "  \"date\": \"2021-05-02\",\n",
    "  \"content\": \"黛玉葬花...\"\n",
    "}\n",
    "\n",
    "PUT /my-index-000001/_doc/3\n",
    "{\n",
    "  \"title\": \"易中天品三国\",\n",
    "  \"date\": \"2021-05-03\",\n",
    "  \"content\": \"草船借箭、空城计...\"\n",
    "}\n",
    "PUT /my-index-000001/_doc/4\n",
    "{\n",
    "  \"title\": \"水浒传\",\n",
    "  \"date\": \"2021-05-03\",\n",
    "  \"content\": \"梁山好汉被团灭...\"\n",
    "}\n",
    "PUT /my-index-000001/_doc/5\n",
    "{\n",
    "  \"title\": \"三国演义\",\n",
    "  \"date\": \"2021-05-03\",\n",
    "  \"content\": \"三国时代，群雄逐鹿...\"\n",
    "}\n",
    "\n",
    "# 默认 查询类型 query-then-fetch\n",
    "GET /my-index-000001/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"query_string\": {\n",
    "      \"query\": \"三国演义\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 调整为 dfs_query_then_fetch\n",
    "# expalin 输出具体的打分计算过程：boost * idf * tf \n",
    "GET /my-index-000001/_search?search_type=dfs_query_then_fetch\n",
    "{\n",
    "  \"query\": {\n",
    "    \"query_string\": {\n",
    "      \"query\": \"三国演义\"\n",
    "    }\n",
    "  },\n",
    "  \"explain\": true\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ad475f",
   "metadata": {},
   "source": [
    "![nested-object-join区别](img/object-nested-join.png)\n",
    "\n",
    "Nested 类型使用场景：\n",
    " 含有 Object 数组。\n",
    " 需要对 Object 中的字段（至少两个及以上）同时进行查询，并维护这种关系。\n",
    "Nested 类型允许相互独立地对对象数组进行索引和查询。如果需要维护数组中每个对象的关系，请使用 nested 数据类型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a4cc704",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Obejct 类型\n",
    "DELETE my_shop\n",
    "#定义 mappingPUT my_shop\n",
    "PUT my_shop\n",
    "{\n",
    "    \"settings\": {\n",
    "        \"index\": {\n",
    "            \"number_of_shards\": 1,\n",
    "            \"number_of_replicas\": 1\n",
    "        }\n",
    "    },\n",
    "    \"mappings\": {\n",
    "        \"properties\": {\n",
    "            \"shopName\": {\n",
    "                \"type\": \"text\",\n",
    "                \"analyzer\": \"ik_smart\"\n",
    "            },\n",
    "            \"shopCode\": {\n",
    "                \"type\": \"keyword\"\n",
    "            },\n",
    "            \"supplier\": {\n",
    "                \"properties\": {\n",
    "                    \"supplier_code\": {\n",
    "                        \"type\": \"keyword\"\n",
    "                    },\n",
    "                    \"supplier_name\": {\n",
    "                        \"type\": \"text\",\n",
    "                        \"analyzer\": \"ik_smart\"\n",
    "                    },\n",
    "                    \"area\": {\n",
    "                        \"properties\": {\n",
    "                            \"province\": {\n",
    "                                \"type\": \"keyword\"\n",
    "                            },\n",
    "                            \"city\": {\n",
    "                                \"type\": \"keyword\"\n",
    "                            }\n",
    "                        }\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "#插入测试数据\n",
    "POST my_shop/_bulk\n",
    "{\"index\":{\"_id\":1}}\n",
    "{\"shopName\":\"苹果热销店铺\",\"shopCode\":\"sc001\",\"supplier\":{\"supplier_code\":\"001\",\"supplier_name\":\"南京农村电商领导者\",\"area\":{\"province\":\"江苏省\",\"city\":\"南京市\"}}}\n",
    "{\"index\":{\"_id\":2}}\n",
    "{\"shopName\":\"美的热销店铺\",\"shopCode\":\"sc002\",\"supplier\":{\"supplier_code\":\"001\",\"supplier_name\":\"南京农村电商领导者\",\"area\":{\"province\":\"江苏省\",\"city\":\"南京市\"}}}\n",
    "{\"index\":{\"_id\":3}}\n",
    "{\"shopName\":\"金沙酒热销店铺\",\"shopCode\":\"sc003\",\"supplier\":{\"supplier_code\":\"002\",\"supplier_name\":\"山东农村电商领导者\",\"area\":{\"province\":\"江苏省\",\"city\":\"南京市\"}}}\n",
    "{\"index\":{\"_id\":4}}\n",
    "{\"shopName\":\"华为热销店铺\",\"shopCode\":\"sc004\",\"supplier\":{\"supplier_code\":\"002\",\"suppliername\":\"山东农村电商领导者\",\"area\":{\"province\":\"山东省\",\"city\":\"青岛市\"}}}\n",
    "\n",
    "#访问时，需要 supplier.supplier_code 指定对象对应的字段\n",
    "POST my_shop/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"supplier.supplier_code\": \"001\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#查询销售区域在南京的所有店铺，通过 supplier.area.city 访问对应的字段值\n",
    "POST my_shop/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"supplier.area.city\": \"南京市\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Join 类型是一种特殊的数据类型，类似父子结构，一个子文档只能有一个父文档，一个父文档可以有多个子文档。在实际使用场景中，推荐使用 Data denormalization 来解决过多关联查询问题，字面解读就是”非规范化存储”，通过冗余存储多字段来达到过多关联的查询问题，避免使用 Join 数据类型，虽然带来了关联的方便性，但是会带来额外的查询开销影响搜索性能。\n",
    "\n",
    "#定义索引，my_goods_sale 为售卖的商品信息，my_goods_comment 为商品的评价信息\n",
    "PUT my_goods_hot_sale\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"my_id\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"my_join_field\": {\n",
    "        \"type\": \"join\",\n",
    "        \"relations\": {\n",
    "          \"my_goods_sale\": \"my_goods_comment\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "#添加商品售卖 ID 为 1 的信息\n",
    "PUT my_goods_hot_sale/_doc/1?refresh\n",
    "{\n",
    "  \"my_id\": \"1\",\n",
    "  \"text\": \"This is a my_goods_sale\",\n",
    "  \"my_join_field\": {\n",
    "    \"name\": \"my_goods_sale\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#添加商品售卖 ID 为 2 的信息\n",
    "PUT my_goods_hot_sale/_doc/2?refresh\n",
    "{\n",
    "  \"my_id\": \"2\",\n",
    "  \"text\": \"This is another my_goods_sale\",\n",
    "  \"my_join_field\": {\n",
    "    \"name\": \"my_goods_sale\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "#添加商品售卖 ID 为 3，父商品为 1，注意父子文档一定要在一个 shard 上\n",
    "PUT my_goods_hot_sale/_doc/3?routing=1&refresh\n",
    "{\n",
    "  \"my_id\": \"3\",\n",
    "  \"text\": \"This is an comment\",\n",
    "  \"my_join_field\": {\n",
    "    \"name\": \"my_goods_comment\",\n",
    "    \"parent\": \"1\"\n",
    "  }\n",
    "}\n",
    "#添加商品售卖 ID 为 4，父商品为 1\n",
    "PUT my_goods_hot_sale/_doc/4?routing=1&refresh\n",
    "{\n",
    "  \"my_id\": \"4\",\n",
    "  \"text\": \"This is another comment\",\n",
    "  \"my_join_field\": {\n",
    "    \"nam\": \"my_goods_comment\",\n",
    "    \"parent\": \"1\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 根据父文档查询子文档：\n",
    "GET my_goods_hot_sale/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"has_parent\": {\n",
    "      \"parent_type\": \"my_goods_sale\",\n",
    "      \"query\": {\n",
    "        \"match\": {\n",
    "          \"text\": \"my_goods_sale\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 根据子文档查询父文档：\n",
    "GET my_goods_hot_sale/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"has_child\": {\n",
    "      \"type\": \"my_goods_comment\",\n",
    "      \"query\": {\n",
    "        \"match_all\": {}\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# Nested 类型\n",
    "\n",
    "PUT goods_info_object\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"goodsName\": {\n",
    "        \"type\": \"text\",\n",
    "        \"analyzer\": \"ik_smart\"\n",
    "      },\n",
    "      \"skuCode\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"brandName\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"shopCode\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"publicPrice\": {\n",
    "        \"type\": \"float\"\n",
    "      },\n",
    "      \"groupPrice\": {\n",
    "        \"properties\": {\n",
    "          \"boxLevelPrice\": {\n",
    "            \"type\": \"keyword\"\n",
    "          },\n",
    "          \"level\": {\n",
    "            \"type\": \"keyword\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#插入测试数据，为了便于阅读 JSON 格式进行了展开\n",
    "POST goods_info_object/_bulk\n",
    "{\n",
    "\"index\": {\"_id\": 1}}\n",
    "{\"goodsName\": \"美国苹果\",\"skuCode\": \"skuCode1\",\"brandName\": \"美国苹果\",\"shopCode\": \"sc00001\",\"publicPrice\": \"8388.88\",\"groupPrice\": [{\"boxLevelPrice\": \"4888.00\",\"level\": \"A\"},\n",
    "{\"boxLevelPrice\": \"6888.00\",\"level\": \"B\"}]}\n",
    "{\"index\": {\"_id\": 2}}\n",
    "{\"goodsName\": \"山东苹果\",\"skuCode\": \"skuCode2\",\"brandName\": \"山东苹果\",\"shopCode\": \"sc00001\",\"publicPrice\": \"7388.88\",\"groupPrice\": [{\"boxLevelPrice\": \"5888.00\",\"level\": \"A\"},{\"boxLevelPrice\": \"4888.00\",\"level\": \"B\"}]}\n",
    "\n",
    "#检索 A 组且价格等于 4888.00 的商品,出现错误，\n",
    "POST goods_info_object/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"bool\": {\n",
    "      \"must\": [\n",
    "        {\n",
    "          \"match\": {\n",
    "            \"groupPrice.level\": \"A\"\n",
    "          }\n",
    "        },\n",
    "        {\n",
    "          \"match\": {\n",
    "            \"groupPrice.boxLevelPrice\": \"4888.00\"\n",
    "          }\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "# 上面查询语句出现不应该的记录，是因为 Elasticsearch 中将 Object 数组打平了做存储导致，在 Elasticsearch中，会将数据做如下存储：\n",
    "#{\n",
    "#    \"goodsName\" : \"山东苹果\",\n",
    "#    \"skuCode\" : \"skuCode2\",\n",
    "#     \"brandName\" : \"山东苹果\",\n",
    "#    \"shopCode\" : \"sc00001\",\n",
    "#    \"publicPrice\" : \"7388.88\",\n",
    "#    \"groupPrice.boxLevelPrice\" :[\"5888.00\",\"4888.00\"],\n",
    "#    \"groupPrice.level\" :[\"A\",\"B\"]\n",
    "#}\n",
    "# 使用nested 类型存储\n",
    "DELETE  goods_info_nested\n",
    "PUT goods_info_nested\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"goodsName\": {\n",
    "        \"type\": \"text\",\n",
    "        \"analyzer\": \"ik_smart\"\n",
    "      },\n",
    "      \"skuCode\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"brandName\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"shopCode\": {\n",
    "        \"type\": \"keyword\"\n",
    "      },\n",
    "      \"publicPrice\": {\n",
    "        \"type\": \"float\"\n",
    "      },\n",
    "      \"groupPrice\": {\n",
    "        \"type\": \"nested\",\n",
    "        \"properties\": {\n",
    "          \"boxLevelPrice\": {\n",
    "            \"type\": \"float\"\n",
    "          },\n",
    "          \"level\": {\n",
    "            \"type\": \"keyword\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "# #插入同样的测试数据\n",
    "POST goods_info_nested/_bulk\n",
    "{\"index\":{\"_id\":1}}\n",
    "{\"goodsName\":\"美国苹果\",\"skuCode\":\"skuCode1\",\"brandName\":\"美国苹果\",\"shopCode\":\"sc00001\",\"publicPrice\":\"8388.88\",\"groupPrice\":[{\"boxLevelPrice\":\"4888.00\",\"level\":\"A\"},{\"boxLevelPrice\":\"6888.00\",\"level\":\"B\"}]}\n",
    "{\"index\":{\"_id\":2}}\n",
    "{\"goodsName\":\"山东苹果\",\"skuCode\":\"skuCode2\",\"brandName\":\"山东苹果\",\"shopCode\":\"sc00001\",\"publicPrice\":\"7388.88\",\"groupPrice\":[{\"boxLevelPrice\":\"5888.00\",\"level\":\"A\"},{\"boxLevelPrice\":\"4888.00\",\"level\":\"B\"}]}\n",
    "\n",
    "GET /goods_info_nested/_search\n",
    "\n",
    "POST goods_info_nested/_search\n",
    "{\n",
    "    \"query\": {\n",
    "        \"nested\": {\n",
    "            \"path\": \"groupPrice\",\n",
    "            \"query\": {\n",
    "                \"bool\": {\n",
    "                    \"must\": [\n",
    "                        {\n",
    "                            \"match\": {\n",
    "                                \"groupPrice.level\": \"A\"\n",
    "                            }\n",
    "                        },\n",
    "                        {\n",
    "                            \"match\": {\n",
    "                                \"groupPrice.boxLevelPrice\": \"4888.00\"\n",
    "                            }\n",
    "                        }\n",
    "                    ]\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "# 在对 Nested Object 进行聚合操作时，我们需要使用到 Nested Aggregation，我们需要聚合查询最大的分组价格( groupPrice )。\n",
    "POST /goods_info_nested/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"goodsName\": \"苹果\"\n",
    "    }\n",
    "  },\n",
    "  \"aggs\": {\n",
    "    \"groupPrice\": {\n",
    "      \"nested\": {\n",
    "        \"path\": \"groupPrice\"\n",
    "      },\n",
    "      \"aggs\": {\n",
    "        \"max_price\": {\n",
    "          \"max\": {\n",
    "            \"field\": \"groupPrice.boxLevelPrice\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2004d874",
   "metadata": {},
   "source": [
    "Elasticsearch 本着让用户方便快捷的使用搜索功能的原则，对数据定义（索引定义）做了高度抽象，尽可能得避免了重复性定义工作，使之更加灵活。\n",
    "Elasticsearch 在这方面做的工作主要体现是索引模板（Index template）和动态映射（Dynamic Mapping）两个功能。索引模板的主要功能，是允许用户在创建索引（index）时，引用已保存的模板来减少配置项。操作的一般过程是先创建索引模板，然后再手动创建索引或保存文档（Document）。而自动创建索引时，索引模板会作为配置的基础作用。对于 X-Pack 的数据流（Data Stream）功能，索引模板用于自动创建后备索引（Backing Indices）。该功能的意义是提供了一种配置复用机制，减少了大量重复作劳动。\n",
    "\n",
    "索引模板：可以根据规则自动创建索引。\n",
    "动态映射（dynamic mapping）：自动将新字段添加到映射中。\n",
    "### 索引模板\n",
    "\n",
    "### 动态映射\n",
    "在 默 认 情 况 下 ， 当 索 引 一 个 文 档 时 有 字 段 是 在 映 射 中 没 有 配 置 的 ， 那 么Elasticsearch 将会根据该属性的类型，自动将其增加到映射中。该功能可以通过配置dynamic 来控制打开。\n",
    "该配置可以接受以下 3 种选择：\n",
    " ture：默认配置，新字段将会自动加入映射中，并自动推断字段的类型。\n",
    " false：新字段不会增加到映射中，因此不能被搜索，但是内容依然会保存在_source 中。如无特殊需要建议都配置为 false，这样可以避免写入流程经过master 节点，从而提高性能。\n",
    " strict：索引文档时如果发现有新字段则报错，整个文档都不会被索引。\n",
    "\n",
    "在使用动态模板时，还有以下几点需要注意。\n",
    " 所有 null 值及空数组属于无效值，不会被任何动态模板匹配，在索引文档时，只有字段第一次有有效值时，才会与各动态模板匹配，找出匹配的模板创建新字段。\n",
    " 规则匹配时，按照动态模板配置的顺序依次对比，使用最先匹配成功的模板，这就意味着如果有字段同时符合 2 个动态模板，那么会使用在 dynamic_templates 数组中靠前的那个。每个动态模板的匹配方式至少应包含 match、path_match 、match_mapping_type 中的一个，unmatch 和 path_unmatch 不能单独使用。\n",
    " mapping 的 dynamic_templates 字段是可以在运行时修改的，每次修改会整体替换 dynamic_templates 的所有值而非追加。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "696425d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建 1 个组件模板 ct1\n",
    "PUT /_component_template/ct1\n",
    "{\n",
    "  \"template\": {\n",
    "    \"settings\": {\n",
    "      \"index.number_of_shards\": 2\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 创建 1 个组件模板 ct2\n",
    "PUT /_component_template/ct2\n",
    "{\n",
    "  \"template\": {\n",
    "    \"settings\": {\n",
    "      \"index.number_of_replicas\": 0\n",
    "    },\n",
    "    \"mappings\": {\n",
    "      \"properties\": {\n",
    "        \"@timestamp\": {\n",
    "          \"type\": \"date\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 创建 1 个索引模板 final-template\n",
    "PUT /_index_template/final-template # 3\n",
    "{\n",
    "  \"index_patterns\": [\n",
    "    \"my-index-*\"\n",
    "  ],\n",
    "  \"composed_of\": [\n",
    "    \"ct1\",\n",
    "    \"ct2\"\n",
    "  ],\n",
    "  \"priority\": 5,\n",
    "  \"template\": {\n",
    "    \"settings\": {\n",
    "      \"index.number_of_replicas\": 1\n",
    "    },\n",
    "    \"mappings\": {\n",
    "      \"properties\": {\n",
    "        \"name\": {\n",
    "          \"type\": \"keyword\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "# 验证创建名为 my-index-00000 的索引使用的配置是如何\n",
    "POST /_index_template/_simulate_index/my-index-00000 \n",
    "\n",
    "GET my-index-00000/_mapping\n",
    "\n",
    "\n",
    "GET /_component_template/template_1?local=false&master_timeout=30s #1\n",
    "GET /_component_template/template_* #2\n",
    "GET /_component_template #3\n",
    "GET /_index_template/test_template #4\n",
    "GET /_index_template/test_* #5\n",
    "GET /_index_template #6\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd5726ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#动态映射\n",
    "# 创建 test-dynamic-mapping\n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"dynamic\": false,\n",
    "    \"properties\": {\n",
    "      \"person\": {\n",
    "        \"dynamic\": true,\n",
    "        \"properties\": {\n",
    "          \"name\": {\n",
    "            \"type\": \"keyword\"\n",
    "          }\n",
    "        }\n",
    "      },\n",
    "      \"company\": {\n",
    "        \"dynamic\": \"strict\",\n",
    "        \"properties\": {\n",
    "          \"company_id\": {\n",
    "            \"type\": \"keyword\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 插入文档\n",
    "PUT test-dynamic-mapping/_doc/1\n",
    "{\n",
    "  \"school\": \"test school\",\n",
    "  \"person\": {\n",
    "    \"name\": \"tom\",\n",
    "    \"age\": \"12\"\n",
    "  },\n",
    "  \"company\": {\n",
    "    \"company_id\": \"c001\"\n",
    "  }\n",
    "}\n",
    "\n",
    "GET test-dynamic-mapping/_mapping\n",
    "\n",
    "# 创建测试索引 dynamic 、numeric_detection、date_detection 自动识别\n",
    "# date_detection 默认为 true，numeric_detection 默认为 false。\n",
    "DELETE test-dynamic-mapping\n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"dynamic\": true,\n",
    "    \"numeric_detection\": true,\n",
    "    \"properties\": {\n",
    "      \"field1\": {\n",
    "        \"type\": \"keyword\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 插入数据\n",
    "PUT test-dynamic-mapping/_doc/1\n",
    "{\n",
    "\"date\":\"2021/05/01\", \n",
    "\"float\":\"1.1\", \n",
    "\"long\":\"1\" \n",
    "}\n",
    "\n",
    "GET test-dynamic-mapping/_mapping\n",
    "\n",
    "# 创建测试索引 指定 date类型动态映射的格式\n",
    "DELETE test-dynamic-mapping\n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"dynamic\": true,\n",
    "    \"dynamic_date_formats\": [\n",
    "      \"MM/dd/yyyy\"\n",
    "    ],\n",
    "    \"properties\": {\n",
    "      \"field1\": {\n",
    "        \"type\": \"keyword\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 插入数据\n",
    "PUT test-dynamic-mapping/_doc/1\n",
    "{\n",
    "\"date\":\"09/25/2015\",\n",
    "\"date1\":\"2015/09/25\"\n",
    "}\n",
    "\n",
    "GET test-dynamic-mapping/_mapping\n",
    "\n",
    "\n",
    "# 动态模板 匹配 mapping_type 为long 转换为integer\n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"dynamic_templates\": [\n",
    "      {\n",
    "        \"test_float\": {\n",
    "          \"match_mapping_type\": \"long\",\n",
    "          \"mapping\": {\n",
    "            \"type\": \"integer\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    ]\n",
    "  }\n",
    "}\n",
    "\n",
    "# 在生产使用中最多的场景，是根据字段的名称进行匹配。匹配 以long_开头，不以_test结尾的属性名 match 和 unmatch 仅作用于最后一级的属性名。\n",
    "DELETE  test-dynamic-mapping\n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "    \"mappings\": {\n",
    "        \"dynamic_templates\": [\n",
    "            {\n",
    "                \"test_float\": {\n",
    "                    \"match\": \"long_*\", \n",
    "                    \"unmatch\": \"*_test\", \n",
    "                    \"mapping\": {\n",
    "                        \"type\": \"long\" \n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "# 使用正则表达式匹配\n",
    "DELETE test-dynamic-mapping \n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "    \"mappings\": {\n",
    "        \"dynamic_templates\": [\n",
    "            {\n",
    "                \"test_float\": {\n",
    "                    \"match_pattern\": \"regex\",\n",
    "                    \"match\": \"^profit_\\\\d+$\",\n",
    "                    \"mapping\": {\n",
    "                        \"type\": \"keyword\"\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "# path_match、path_unmatch \n",
    "# 如下模板表示设置 person内嵌对象除了 age 外其它所有以long_开头的字段新增时类型设为 text。\n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "    \"mappings\": {\n",
    "        \"dynamic_templates\": [\n",
    "            {\n",
    "                \"test_float\": {\n",
    "                    \"match_pattern\": \"long_\",\n",
    "                    \"path_match\": \"person.*\",\n",
    "                    \"path_unmatch\": \"*.age\",\n",
    "                    \"mapping\": {\n",
    "                        \"type\": \"text\"\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "DELETE test-dynamic-mapping\n",
    "PUT test-dynamic-mapping\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"dynamic_templates\": [\n",
    "      {\n",
    "        \"named_analyzers\": {\n",
    "          \"match_mapping_type\": \"string\",\n",
    "          \"match\": \"*\",\n",
    "          \"mapping\": {\n",
    "            \"type\": \"text\",\n",
    "            \"analyzer\": \"{name}\"\n",
    "          }\n",
    "        }\n",
    "      },\n",
    "      {\n",
    "        \"no_doc_values\": {\n",
    "          \"match_mapping_type\": \"*\",\n",
    "          \"mapping\": {\n",
    "            \"type\": \"{dynamic_type}\",\n",
    "            \"doc_values\": false\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    ]\n",
    "  }\n",
    "}\n",
    "\n",
    "PUT test-dynamic-mapping/_doc/1\n",
    "{\n",
    "  \"english\": \"Some English text\",\n",
    "  \"count\": 5\n",
    "}\n",
    "\n",
    "GET test-dynamic-mapping\n",
    "\n",
    "# 将映射 test-dynamic-mapping 原来的动态模板配置删除，并配一个名为 newTemplate 的动态模板。替换所有值，并不是追加\n",
    "PUT test-dynamic-mapping/_mapping\n",
    "{\n",
    "  \"dynamic_templates\": [\n",
    "    {\n",
    "      \"newTemplate\": {\n",
    "        \"match\": \"abc*\",\n",
    "        \"mapping\": {\n",
    "          \"type\": \"keyword\"\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34785247",
   "metadata": {},
   "source": [
    "查询模板\n",
    "\n",
    "Elasticsearch 允许使用模板语言 mustache 来预设搜索逻辑，在实际搜索时，通过参数中的键值，对来替换模板中的占位符，最终完成搜索。该方式将搜索逻辑封闭在Elasticsearch 中，可以使下游服务，在不知道具体搜索逻辑的情况下完成数据检索。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "075455f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 查询模板\n",
    "\n",
    "POST _scripts/testSearchTemplate\n",
    "{\n",
    "  \"script\": {\n",
    "    \"lang\": \"mustache\",\n",
    "    \"source\": {\n",
    "      \"query\": {\n",
    "        \"term\": {\n",
    "          \"FlightNum\": {\n",
    "            \"value\": \"{{FlightNum}}\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "GET kibana_sample_data_flights/_search/template\n",
    "{\n",
    "  \"id\": \"testSearchTemplate\",\n",
    "  \"params\": {\n",
    "    \"FlightNum\": \"9HY9SWR\"\n",
    "  }\n",
    "}\n",
    "\n",
    "GET _scripts/<templateId>\n",
    "DELETE _scripts/<templateId>\n",
    "\n",
    "# 尝试 搜索模板 是否达到搜索效果\n",
    "GET _render/template\n",
    "{\n",
    "\"source\":\n",
    "\"\"\"{\"query\": {\"term\": {\"FlightNum\": {\"value\": \"{{FlightNum}}\"}}}}\"\"\" ,\n",
    "\"params\":{ \"FlightNum\": \"9HY9SWR\"}\n",
    "}\n",
    "\n",
    "\n",
    "GET _render/template\n",
    "{\n",
    "    \"source\":\"\"\"{\"query\": {\"term\": {\"FlightNum\": {{#toJson}}FlightNum{{/toJson}} }}\"\"\", \n",
    "    \"params\":{ \"FlightNum\": {\"value\":\"9HY9SWR\"}}\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "GET _render/template\n",
    "{\n",
    "\"source\":{\"query\":{\"term\": {\"FlightNum\": \"{{#join delimiter='||'}}FlightNums{{/join delimiter='||'}\",\"DestCountry\":\"{{DestCountry}}{{^DestCountry}}AU{{/DestCountry}}\",\"Dest\": \"{{#url}}{{Dest}}{{/url}}\"}}},\"params\":{\"FlightNums\": [\"9HY9SWR\",\"adf2c1\"],\"Dest\":\"http://www.baidu.com\"}\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edcd38f",
   "metadata": {},
   "source": [
    "索引别名是一个非常好的”工具“，他可以帮助解决以下问题：\n",
    " 如果对写入 Elasticsearch 的数据进行极少的修改，索引别名+ Rollover 可以很好控制每个索引的大小，零停机切换索引；合适的索引大小可以提升数据的查询性能，数据恢复性能。\n",
    " 解耦 client 与索引的强耦合，Elasticsearch 维护人员可以对索引有更灵活的操作空间，且让用户侧无感知。\n",
    " 结合 Reindex 可以很方便的完成索引重建。\n",
    " 过滤别名和路由别名可以在一定程度上帮助提升查询性能。\n",
    " 一个绑定多个索引的别名，如果要查询一次查询这多个索引，别名可以使 uri 变的简洁。\n",
    "\n",
    "依据不同的使用场景，我们可以简单把别名分为三类：\n",
    "（1）索引别名\n",
    "（2）过滤别名\n",
    "（3）路由别名\n",
    "\n",
    "过滤别名\n",
    "一种创建同一索引的不同”视图“的简便方法。\n",
    "通过将过滤条件绑定到对应别名，使用不同别名即获取满足不通条件的数据；使用Query DSL 定义过滤器。\n",
    "使用过滤别名，必须得保证过滤字段存在，因此提前创建好索引，并设定该字段的\n",
    "schema。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06a7accd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#为 users1 表创建 index-alias-name1 别名\n",
    "PUT /users1/_alias/index-alias-name1\n",
    "#为 users1,users2 表创建 index-alias-name2 别名\n",
    "PUT /users1,users2/_alias/index-alias-name2\n",
    "#为 users 开头的索引创建 index-alias-name3 别名\n",
    "PUT /users*/_alias/index-alias-name3\n",
    "\n",
    "#为 users1 索引添加具有路由和过滤功能的别名 routing-filter-index-alias\n",
    "PUT users1/_alias/routing-filter-index-alias\n",
    "{\n",
    "  \"routing\": \"12\",\n",
    "  \"filter\": {\n",
    "    \"term\": {\n",
    "      \"user_id\": 12\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#为 users1 索引添加路由别名 routing-index-alias，路由计算值为 12\n",
    "PUT users1/_alias/routing-index-alias\n",
    "{\n",
    "  \"routing\": \"12\"\n",
    "}\n",
    "\n",
    "#为 users1 索引添加过滤别名 filter-index-alias，过滤 user_id 为 12\n",
    "PUT users1/_alias/filter-index-alias\n",
    "{\n",
    "  \"filter\": {\n",
    "    \"term\": {\n",
    "      \"user_id\": 12\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#通过别名，写入数据\n",
    "PUT index-alias-name1/_bulk?refresh=true\n",
    "{\n",
    "  \"index\": {}\n",
    "}\n",
    "{\n",
    "  \"user_id\": \"tom123456-user1\"\n",
    "}\n",
    "\n",
    "#通过路由别名，写入数据\n",
    "PUT routing-index-alias/_bulk?refresh=true\n",
    "{\n",
    "  \"index\": {}\n",
    "}\n",
    "{\n",
    "  \"user_id\": \"kimchy123456-routing\"\n",
    "}\n",
    "\n",
    "#通过索引，写入数据\n",
    "PUT users2/_bulk?refresh=true\n",
    "{\n",
    "  \"index\": {}\n",
    "}\n",
    "{\n",
    "  \"user_id\": \"kimchy123456-user2\"\n",
    "}\n",
    "{\n",
    "  \"index\": {}\n",
    "}\n",
    "{\n",
    "  \"user_id\": \"12\"\n",
    "}\n",
    "\n",
    "#以下三条查询语句的结果是等价的，返回 2 条数据\n",
    "GET index-alias-name1/_search\n",
    "GET users1/_search\n",
    "GET routing-index-alias/_search\n",
    "\n",
    "#以下三条查询语句的结果是等价的(如果没有其他 users 开头的索引)，返回四条数据\n",
    "GET users1,users2/_search\n",
    "GET users*/_search\n",
    "GET index-alias-name2/_search\n",
    "\n",
    "#查询 user2 的索引，返回 2 条数据\n",
    "GET users2/_search\n",
    "#返回值为空，因为索引 users1 插入的文档没有 user_id 值为 2 的\n",
    "GET filter-index-alias/_search\n",
    "# 返回值为一条，因为索引 users2 有一条 user_id 值为 2 的文档\n",
    "GET index-alias-name3/_search\n",
    "\n",
    "#批量创建别名\n",
    "POST /_aliases\n",
    "{\n",
    "  \"actions\": [\n",
    "    {\n",
    "      \"add\": {\n",
    "        \"index\": \"test1\",\n",
    "        \"alias\": \"alias1\"\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"add\": {\n",
    "        \"index\": \"test2\",\n",
    "        \"alias\": \"alias1\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "# 别名重命名\n",
    "POST /_aliases\n",
    "{\n",
    "  \"actions\": [\n",
    "    {\n",
    "      \"remove\": {\n",
    "        \"index\": \"test1\",\n",
    "        \"alias\": \"alias1\"\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"add\": {\n",
    "        \"index\": \"test1\",\n",
    "        \"alias\": \"alias2\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\n",
    "# 将一个别名关联多个索引\n",
    "POST /_aliases\n",
    "{\n",
    "  \"actions\": [\n",
    "    {\n",
    "      \"add\": {\n",
    "        \"index\": \"test1\",\n",
    "        \"alias\": \"alias1\"\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"add\": {\n",
    "        \"index\": \"test2\",\n",
    "        \"alias\": \"alias1\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "# 等效于上面的方式\n",
    "POST /_aliases\n",
    "{\n",
    "  \"actions\": [\n",
    "    {\n",
    "      \"add\": {\n",
    "        \"indices\": [\n",
    "          \"test1\",\n",
    "          \"test2\"\n",
    "        ],\n",
    "        \"alias\": \"alias1\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\n",
    "\n",
    "# 错误的创建了一个名称为 test 的索引，而实际需要的索引名称为 test_2，但是已经有数据往索引里面写入数据了；为了解决这个问题，首先创建正确的索引名称，然后用一个原子操作，将 test 别名绑定 test_2，同时删除索引 test。\n",
    "# 创建索引 test\n",
    "PUT test\n",
    "# 创建索引 test_2\n",
    "PUT test_2\n",
    "POST /_aliases\n",
    "{\n",
    "  \"actions\": [\n",
    "    {\n",
    "      \"add\": {\n",
    "        \"index\": \"test_2\",\n",
    "        \"alias\": \"test\"\n",
    "      }\n",
    "    },\n",
    "    {\n",
    "      \"remove_index\": {\n",
    "        \"index\": \"test\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba356f65",
   "metadata": {},
   "source": [
    "Reindex\n",
    "\n",
    "此外，使用 Reindex 需要注意以下几点：\n",
    " 源和目的不能相同，比如不能将数据流 Reindex 给它自身。\n",
    " 源索引的文档中 _source 字段必须开启。\n",
    " Reindex 不会复制源的 setting 和源所匹配的模板，因此在调用 _reindex 前，你需要设置好目的索引 (action.auto_create_index 为 false 或者 -.* 时)。\n",
    " 目标索引的 mapping，主分片数，副本数等推荐提前配置。\n",
    "Reindex 的主要场景：\n",
    " 集群升级：将数据从旧集群远程 Reindex 到新集群\n",
    " 索引备份\n",
    " 数据重构"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "173f56dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#重建索引 参数较多，用到再查\n",
    "POST _reindex?wait_for_active_shards=2&timeout=5s\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"reindex_index-name-1\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"reindex_index-name-2\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#异步执行 reindex 任务\n",
    "POST _reindex?wait_for_completion=false\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"reindex_index-name-1\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"reindex_index-name-2\"\n",
    "  }\n",
    "}\n",
    "#返回立即返回的任务 id\n",
    "#{\n",
    "#\"task\" : \"ydZx8i8HQBe69T4vbYm30g:20987804\"\n",
    "#}\n",
    "#查看任务的运行情况\n",
    "GET _tasks/ydZx8i8HQBe69T4vbYm30g:20987804\n",
    "\n",
    "\n",
    "# 多源 重建索引\n",
    "#!/bin/bashfor index in i1 i2 i3 i4 i5; do\n",
    "#curl -H Content-Type:application/json -XPOST localhost:9200/_reindex?pretty -d'{\n",
    "#\"source\":{\n",
    "#  \"index\": \"'$index'\"\n",
    "#},\n",
    "#  \"dest\": {\n",
    "  #\"index\": \"'$index'-reindexed\"\n",
    "#}}'done\n",
    "\n",
    "\n",
    "# Reindex 支持切片 scroll 以并行化重新索引过程，从而提高 Reindex 的效率。注意：如果源索引是在远程的 Elasticsearch 集群，是不支持手动或自动切片的。\n",
    "# 手动分片：通过为每个请求提供切片 ID 和切片总数。\n",
    "POST _reindex\n",
    "{\n",
    "    \"source\": {\n",
    "        \"index\": \"my-index-000001\",\n",
    "        \"slice\": {\n",
    "            \"id\": 1,\n",
    "            \"max\": 2\n",
    "        }\n",
    "    },\n",
    "    \"dest\": {\n",
    "        \"index\": \"my-new-index-000001\"\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\",\n",
    "    \"slice\": {\n",
    "      \"id\": 1,\n",
    "      \"max\": 2\n",
    "    }\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  }\n",
    "}\n",
    "# 验证步骤\n",
    "#避免还没有形成 segments，文档不可见\n",
    "GET _refresh\n",
    "#查看文档的个数\n",
    "GET my-new-index-000001/_count\n",
    "#或者\n",
    "POST my-new-index-000001/_search?size=0&filter_path=hits.total\n",
    "\n",
    "\n",
    "#自动分片\n",
    "# 还可以使用 Sliced scroll 基于文档_id 进行切片，让_reindex 自动并行化；通过设定 slices 参数的值来实现。\n",
    "POST _reindex?slices=5&refresh\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# # 由源码可知，slices 实际被修改为 0\n",
    "# 设置 slices 为 auto 会让 Elasticsearch 选择要使用的切片数。此设置将每个分片使用一个切片，直到达到某个限制。如果有多个源，它将基于分片数量最少的索引或Backing index 确定切片数。\n",
    "POST _reindex?slices=auto&refresh\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  }\n",
    "}\n",
    "\n",
    "## 选择切片数的注意事项： 当 slices 的数目等于索引的数目时，查询性能是最优的。设置 slices 高于分片数通常不会提高效率，反而会增加开销（CPU，磁盘 IO 等）。\n",
    "# 索引性能会在可用资源与切片数之间线性地缩放。\n",
    "# 查询或索引性能在运行时是否占主导地位，取决于重新索引的文档和集群资源。\n",
    "\n",
    "# 重新索引的路由\n",
    "# 默认情况下，_reindex 使用滚动批处理 1000。你可以使用元素中的 size 字段更改批处理大小：\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"source_index\",\n",
    "    \"size\": 500,\n",
    "    \"query\": {\n",
    "      \"match\": {\n",
    "        \"company\": \"cat\"\n",
    "      }\n",
    "    }\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"dest_index\",\n",
    "    \"routing\": \"=cat\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#重索引使用预处理 Pipeline\n",
    "# 重索引也可以使用 ingest pipeline 的特性，来富化数据\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"source\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"dest\",\n",
    "    \"pipeline\": \"some_ingest_pipeline\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "## 实战示例\n",
    "#基于查询重新索引文档\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\",\n",
    "    \"query\": {\n",
    "      \"term\": {\n",
    "        \"user.id\": \"kimchy\"\n",
    "      }\n",
    "    }\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#基于 max_docs 重新索引文档\n",
    "POST _reindex\n",
    "{\n",
    "  \"max_docs\": 1,\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#基于多源重新索引\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": [\n",
    "      \"my-index-000001\",\n",
    "      \"my-index-000002\"\n",
    "    ]\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000002\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#选择字段重新索引\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\",\n",
    "    \"_source\": [\n",
    "      \"user.id\",\n",
    "      \"_doc\"\n",
    "    ]\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 通过重新索引修改文档中字段名字\n",
    "#你想把字段名 flag 替换成 tag，\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  },\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.tag = ctx._source.remove(\\\"flag\\\")\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 重新索引每日索引\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"metricbeat-*\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"metricbeat\"\n",
    "  },\n",
    "  \"script\": {\n",
    "    \"lang\": \"painless\",\n",
    "    \"source\": \"ctx._index = 'metricbeat-' + (ctx._index.substring('metricbeat-'.length(),ctx._index.length())) + '-1'\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 提取源的随机子集\n",
    "POST _reindex\n",
    "{\n",
    "  \"max_docs\": 10,\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\",\n",
    "    \"query\": {\n",
    "      \"function_score\": {\n",
    "        \"random_score\": {},\n",
    "        \"min_score\": 0.9\n",
    "      }\n",
    "    }\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 重新索引时修改文档\n",
    "# 像_update_by_query 一样，_reindex 支持使用 script 修改文档；不同的是，reindex 中使用脚本可以修改文档的元数据。\n",
    "\n",
    "POST _reindex\n",
    "{\n",
    "  \"source\": {\n",
    "    \"index\": \"my-index-000001\"\n",
    "  },\n",
    "  \"dest\": {\n",
    "    \"index\": \"my-new-index-000001\",\n",
    "    \"version_type\": \"external\"\n",
    "  },\n",
    "  \"script\": {\n",
    "    \"source\": \"if (ctx._source.foo == 'bar') {ctx._version++; ctx._source.remove('foo')}\",\n",
    "    \"lang\": \"painless\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 远程重新索引\n",
    "# reindex.remote.whitelist: \"otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*\"\n",
    "POST _reindex\n",
    "{\n",
    "    \"source\": {\n",
    "        \"remote\": {\n",
    "            \"host\": \"http://otherhost:9200\",\n",
    "            \"username\": \"user\",\n",
    "            \"password\": \"pass\",\n",
    "            \"socket_timeout\": \"1m\",\n",
    "            \"connect_timeout\": \"10s\"\n",
    "        },\n",
    "        \"index\": \"my-index-000001\",\n",
    "        \"query\": {\n",
    "            \"match\": {\n",
    "                \"test\": \"data\"\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \"dest\": {\n",
    "        \"index\": \"my-new-index-000001\"\n",
    "    }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90fee620",
   "metadata": {},
   "source": [
    "Rollover index \n",
    "了解 Elasticsearch 的同学应该都知道，索引的主分片在设定之后，改变（shrink ，split，reindex）主分片数目的成本相当大；因此在设计之初，一定要规划好索引的分片数目。如果集群的中节点数目固定，且写入的数据不会再有更新操作或者更新操作极其少；可以使用 Rollover index 的方式来限制每个索引的大小。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898c3232",
   "metadata": {},
   "source": [
    "POST /<rollover-target>/_rollover/<target-index>?wait_for_active_shards=<number>\n",
    "POST /<rollover-target>/_rollover?wait_for_active_shards=<number>\n",
    "\n",
    "PUT /logs-1 #备注 1\n",
    "{\n",
    "  \"aliases\": {\n",
    "    \"logs_write\": {}\n",
    "  }\n",
    "}\n",
    "\n",
    "# Add > 2 documents to logs-1\n",
    "PUT logs-1/_bulk\n",
    "{\n",
    "  \"index\": {}\n",
    "}\n",
    "{\n",
    "  \"user.id\": \"kimchy\"\n",
    "}\n",
    "{\n",
    "  \"index\": {}\n",
    "}\n",
    "{\n",
    "  \"user.id\": \"tom\"\n",
    "}\n",
    "\n",
    "POST /logs_write/_rollover #备注 2\n",
    "{\n",
    "  \"conditions\": {\n",
    "    \"max_age\": \"7d\",\n",
    "    \"max_docs\": 2,\n",
    "    \"max_size\": \"5gb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "GET /logs-000002/_search\n",
    "\n",
    "# 查看别名与索引的绑定关系\n",
    "GET _cat/aliases/logs_write?v\n",
    "\n",
    "\n",
    "\n",
    "# 基于数据流滚动\n",
    "\n",
    "PUT _index_template/template\n",
    "{\n",
    "\"index_patterns\": [\"my-data-stream*\"],\n",
    "\"data_stream\": { }\n",
    "}\n",
    "\n",
    "PUT /_data_stream/my-data-stream # 备注 1\n",
    "\n",
    "GET /_data_stream/my-data-stream\n",
    "{\"data_streams\" : [{\"name\":\"my-data-stream\",\"timestamp_field\" : {\"name\":\"@timestamp\"},\n",
    "\"indices\" : [{\"index_name\" : \".ds-my-data-stream-000001\",\"index_uuid\" : \"Vir6yRm4S42k8n22mZ5YBw\"}],\"generation\" : 1,\"status\" : \"GREEN\",\"template\" : \"my-index-template\",\n",
    "\"ilm_policy\" : \"my-lifecycle-policy\"}]}\n",
    "\n",
    "# Add > 2 documents to my-data-stream\n",
    "#为了能够实时的看到效果插入数据时，加上 refresh 参数\n",
    "PUT my-data-stream/_bulk?refresh\n",
    "{ \"create\":{ } }\n",
    "{ \"@timestamp\": \"2099-05-06T16:21:15.000Z\", \"message\": \"192.0.2.42 - - [06/May/2099:16:2\n",
    "1:15 +0000] \\\"GET /images/bg.jpg HTTP/1.0\\\" 200 24736\" }\n",
    "{ \"create\":{ } }\n",
    "{ \"@timestamp\": \"2099-05-06T16:25:42.000Z\", \"message\": \"192.0.2.255 - - [06/May/2099:16:\n",
    "25:42 +0000] \\\"GET /favicon.ico HTTP/1.0\\\" 200 3638\" }\n",
    "{ \"create\":{ } }\n",
    "{ \"@timestamp\": \"2099-05-06T16:27:42.000Z\", \"message\": \"192.0.2.255 - - [06/May/2099:16:\n",
    "25:42 +0000] \\\"GET /favicon.ico HTTP/1.0\\\" 200 3638\" }\n",
    "\n",
    "POST /my-data-stream/_rollover #备注 1\n",
    "{\"conditions\" : {\"max_age\":\"7d\",\"max_docs\": 2,\"max_size\": \"5gb\"}}\n",
    "\n",
    "\n",
    "\n",
    "#1、 数据流必须要提前设定一个索引模板，否则无法创建数据流。\n",
    "PUT _index_template/template\n",
    "{\n",
    "\"index_patterns\": [\"my-data-stream*\"],\n",
    "\"data_stream\": { }\n",
    "}\n",
    "\n",
    "# 2、创建数据流\n",
    "PUT /_data_stream/my-data-stream # 备注 1\n",
    "\n",
    "GET /_data_stream/my-data-stream\n",
    "\n",
    "# 3、插入数据\n",
    "# Add > 2 documents to my-data-stream\n",
    "#为了能够实时的看到效果插入数据时，加上 refresh 参数\n",
    "PUT my-data-stream/_bulk?refresh\n",
    "{ \"create\":{ } }\n",
    "{ \"@timestamp\": \"2099-05-06T16:21:15.000Z\", \"message\": \"192.0.2.42 - - [06/May/2099:16:2\n",
    "1:15 +0000] \\\"GET /images/bg.jpg HTTP/1.0\\\" 200 24736\" }\n",
    "{ \"create\":{ } }\n",
    "{ \"@timestamp\": \"2099-05-06T16:25:42.000Z\", \"message\": \"192.0.2.255 - - [06/May/2099:16:\n",
    "25:42 +0000] \\\"GET /favicon.ico HTTP/1.0\\\" 200 3638\" }\n",
    "{ \"create\":{ } }\n",
    "{ \"@timestamp\": \"2099-05-06T16:27:42.000Z\", \"message\": \"192.0.2.255 - - [06/May/2099:16:\n",
    "25:42 +0000] \\\"GET /favicon.ico HTTP/1.0\\\" 200 3638\" }\n",
    "\n",
    "# 4、执行滚动\n",
    "POST /my-data-stream/_rollover #备注 1\n",
    "{\n",
    "  \"conditions\": {\n",
    "    \"max_age\": \"7d\",\n",
    "    \"max_docs\": 2,\n",
    "    \"max_size\": \"5gb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "#目标索引设定 setting\n",
    "PUT /logs-000001\n",
    "{\n",
    "  \"aliases\": {\n",
    "    \"logs_write\": {}\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "POST /logs_write/_rollover\n",
    "{\n",
    "  \"conditions\": {\n",
    "    \"max_age\": \"7d\",\n",
    "    \"max_docs\": 2,\n",
    "    \"max_size\": \"5gb\"\n",
    "  },\n",
    "  \"settings\": {\n",
    "    \"index.number_of_shards\": 2\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "# my_alias 绑定的索引索引名称不满足 indexName-<number>格式\n",
    "#必须指定索引 my_new_index_name\n",
    "POST /my_alias/_rollover/my_new_index_name\n",
    "{\n",
    "  \"conditions\": {\n",
    "    \"max_age\": \"7d\",\n",
    "    \"max_docs\": 2,\n",
    "    \"max_size\": \"5gb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "## 基于date math 滚动\n",
    "#如果<rollover-target>是索引别名，使用 date math 根据索引滚动的当前日期来命名滚动索引；在某些场景中是非常有用的，比如定时按天删除索引、按天查询数据等。\n",
    "#rollover API 支持 date math，但是要求索引名称必须以日期-<number>结尾，\n",
    "# PUT /<logs-{now/d}-1> with URI encoding:\n",
    "\n",
    "PUT /%3Clogs_%7Bnow%2Fd%7D-1%3E #备注 1\n",
    "{\n",
    "  \"aliases\": {\n",
    "    \"logs_write\": {\"is_write_index\": true}\n",
    "  }\n",
    "}\n",
    "\n",
    "GET /_alias\n",
    "\n",
    "PUT /logs_write/_doc/1\n",
    "{\n",
    "  \"message\": \"adummy log\"\n",
    "}\n",
    "\n",
    "POST logs_write/_refresh\n",
    "\n",
    "# 立即执行 _rollover\n",
    "POST /logs_write/_rollover #备注 2\n",
    "{\n",
    "  \"conditions\": {\n",
    "    \"max_docs\": 1,\n",
    "    \"max_age\": \"1d\",\n",
    "    \"max_size\": \"5gb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "# Dry run\n",
    "# rollover API 支持 dry_run 模式，该模式主要用于条件检测。比如想检测索引是否满足设定的条件完成滚动。\n",
    "POST /logs_write/_rollover?dry_run\n",
    "{\n",
    "  \"conditions\": {\n",
    "    \"max_age\": \"7d\",\n",
    "    \"max_docs\": 1000,\n",
    "    \"max_size\": \"5gb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "# 基于write index 滚动\n",
    "# 基于 write index 滚动主要是为了解决，使用别名查询能够获取到之前创建的索引数据，以及同一别名绑定多个索引滚动切换歧义问题。同一个别名具有查询和写入两种特性。\n",
    "\n",
    "#创建一个索引，绑定别名 logs，并标注 is_write_index 为 true，即通过别名写入数据，实际是\n",
    "#写入到 my_logs_index-000001 表中\n",
    "PUT my_logs_index-000001\n",
    "{\n",
    "  \"aliases\": {\n",
    "    \"logs\": {\n",
    "      \"is_write_index\": true\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#写入一个文档\n",
    "PUT logs/_doc/1\n",
    "{\"message\":\"adummy log\"}\n",
    "\n",
    "POST logs/_refresh\n",
    "\n",
    "#刷新，生成一个 segements,使文档变得可搜索POST logs/_refresh\n",
    "#执行滚动操作 滚动产生的新索引 对应的别名为search_all\n",
    "POST /logs/_rollover #备注 1\n",
    "{\n",
    "  \"aliases\": {\n",
    "    \"search_all\": {}\n",
    "  },\n",
    "  \"conditions\": {\n",
    "    \"max_docs\": \"1\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#再次基于别名写入\n",
    "PUT logs/_doc/2\n",
    "{\"message\":\"a newer log\"}\n",
    "\n",
    "GET _alias/logs\n",
    "\n",
    "\n",
    "\n",
    "##  基于ILM 实现自动Rollover\n",
    "# 我们定义了一个名称为 rollover-test-all 的生命周期管理策略，其中滚动条件为 max_size，为 20gb,max_age 为 1 天，max_docs 为 2；\n",
    "\n",
    "PUT _index_template/iml-rollover_template\n",
    "{\n",
    "  \"index_patterns\": [\n",
    "    \"iml-rollover*\"\n",
    "  ],\n",
    "  \"template\": {\n",
    "    \"settings\": {\n",
    "      \"lifecycle\": {\n",
    "        \"name\": \"rollover-test-all\",\n",
    "        \"rollover_alias\": \"iml-rollover_write_alias\"\n",
    "      },\n",
    "      \"refresh_interval\": \"2s\",\n",
    "      \"number_of_shards\": \"1\",\n",
    "      \"number_of_replicas\": \"0\"\n",
    "    },\n",
    "    \"mappings\": {\n",
    "      \"_source\": {\n",
    "        \"enabled\": true\n",
    "      },\n",
    "      \"properties\": {\n",
    "        \"name\": {\n",
    "          \"type\": \"keyword\"\n",
    "        }\n",
    "      }\n",
    "    },\n",
    "    \"aliases\": {\n",
    "      \"iml-rollover_alias\": {}\n",
    "    }\n",
    "  },\n",
    "  \"priority\": 500,\n",
    "  \"_meta\": {\n",
    "    \"description\": \"my custom rollover test\"\n",
    "  }\n",
    "}\n",
    "\n",
    "PUT iml-rollover-000001\n",
    "{\n",
    "  \"aliases\": {\n",
    "    \"iml-rollover_write_alias\": {\n",
    "      \"is_write_index\": true\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "PUT iml-rollover_write_alias/_bulk?refresh=true\n",
    "{\"index\":{}}\n",
    "{\"name\":\"kimchy\"}\n",
    "{\"index\":{}}\n",
    "{\"name\":\"tom\"}\n",
    "\n",
    "GET _cat/indices/iml-rollover-*?v&h=health,index,docs.count\n",
    "\n",
    "GET _cat/aliases/iml-rollover_*?format=json\n",
    "\n",
    "\n",
    "# 分页搜索\n",
    "#  通过 from 、size 进行分页：支持跳页，不适合深分页\n",
    "# 通过 scroll 拉取大量数据：不支持跳页，适合拉取大量数据，不适合大量并发\n",
    "# 通过 search_after 拉取大量数据：不支持跳页，适合拉取大量数据， 推荐\n",
    "#   search_after 需要提供一个主键字段进行排序，默认为 _shard_doc，它是 shard index 与 Lucene 内部 ID 的组合值。在服务端保存的上下文要比 scroll 小，目前官方推荐使用 search_after 代替 scroll。\n",
    "\n",
    "# from + size 解决场景：这种最经典的翻页模式是为了解决用户对于 TopN 的需求，用户希望找到 TopN 个最匹配的文档。其使用方式类似 SQL 中的 LIMIT 关键字\n",
    "GET /_search?size=5\n",
    "GET /_search?size=5&from=5\n",
    "GET /_search?size=5&from=10\n",
    "\n",
    "#分页搜索实现方式是：\n",
    "# 每个分片各自查询的时候先构建 from+size 的优先队列，然后将所有的文档 ID 和排序值返回给协调节点。\n",
    "# 协调节点创建 size 为 number_of_shards * (from + size) 的优先队列，对数据节点的返回结果进行合并，取全局的 from+size 返回给客#户端。\n",
    "\n",
    "\n",
    "# scroll 解决场景：scroll 可以用于拉取全量数据，他的工作模式不需要像 from + size 一样全局排序，因此没有深分页的代价，例如 reindex 本质上就是 scroll+bulk。使用 scroll 可以简单的在查询语句中添加 scroll 参数：\n",
    "  # 1m 代表 服务端保存缓存数据1分钟\n",
    "POST /my-index-000001/_search?scroll=1m\n",
    "\n",
    "POST /my-index-000001/_search/scroll\n",
    "{\n",
    "  \"scroll\": \"1m\",\n",
    "  \"scroll_id\": \"rrapu312DsdSAhKKv0ilQJ5zTPqvZ7xNbZVxvsNW7CY8CgiZzG9M1IaOHEkN5Sx3z1JCDJolAT8z3hqCPGtUZWjmbP9qegi2jFdOkdWXl4LJYeVVRG5E91nJQkbJfeeVLQ6tRuDGtZkyP4g9lxuDhSNwRVYgOyWCN4IYPCr2AjTfI1Y38dqCasX5IUc9h9i9XoRATXc1UNWdpU9MO2A9ibH6TtG7oFhgAkt7vqISre6VWKiZ4d1EdKBvJFc26Dht4P9BvYu5BpSOrZZbti1Q\"\n",
    "}\n",
    "\n",
    "\n",
    "# search_after 解决场景：如果说 scroll 是把上下文保存在服务端，而 search_after 要求数据中存在一个无重复，可以用于排序的字段，需要客户端每次传入上次查到的最后结果，然后获取其随后的数据。\n",
    "\n",
    "# 获取pit id ，后续查询带这个id ，为了防止这种情况，需要在请求中加一个参数来设置当前的索引状态保留时间。\n",
    "POST /my-index-000001/_pit?keep_alive=1m\n",
    "\n",
    "GET /_search\n",
    "{\n",
    "  \"size\": 10000,\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"user.id\": \"elkbee\"\n",
    "    }\n",
    "  },\n",
    "  \"pit\": {\n",
    "    \"id\": \"UmNDdHdpUG9MUlAwOFZsRGhRNGVlU0ljUXFGdk10MGZXZHpGUjNxaUVTcXcxNTFWVk93M1JPdmxjWXV6V05reXR4dEduYUQ5QkR6MDJWZWNacXBMVm82TXA4M1pVaUtsYzdCVWJjdzVsUnBRRm9rbFViTDd0bk52aWRtTnZtNmo3dUt0Z09JNTNjSjM3QnNvNnlCQXl3TnVDNEV5NzJFUVVXbEZkQjhRNHVDRzEwOGVKc3ZYdHNkWGJ3a1d3U2dvZzRiYWhnSlpPdlh4UklRSWZSOTZ6VmRZN0hkR09TSFJiV2hCUGFHaG5RUU5wUG1DRlM4Rzhua2l1MzhkYjFVSnZjMWlFQzRPaFdVREY2a0lWNk5tVDdaNGZXQ1Z4SFZTT1NMRGZMVjRQaXlHMzFxbFA5WGVwNkp0bUd6cWhvQU9iaTFOeENTSHI5ZnpJdGRRRDBteQ==\",\n",
    "    \"keep_alive\": \"1m\"\n",
    "  },\n",
    "  \"sort\": [\n",
    "    {\n",
    "      \"@timestamp\": \"asc\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\n",
    "# search after  为 查询结果的sort 序号值， 查询排序之后的数据\n",
    "\n",
    "GET /_search\n",
    "{\n",
    "  \"size\": 10000,\n",
    "  \"query\": \"\",\n",
    "  \"pit\": {\n",
    "    \"id\": \"\",\n",
    "    \"keep_alive\": \"\"\n",
    "  },\n",
    "  \"sort\": [\n",
    "    {\n",
    "      \"@timestamp\": \"asc\"\n",
    "    }\n",
    "  ],\n",
    "  \"search_after\": [\n",
    "    4294967298\n",
    "  ]\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b467ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "Ingest Pipeline\n",
    "用于预处理数据，由 Elasticsearch Ingest Node 节点负责运行处理，如需要系统性能提升可单独部署 Ingest Node 节点。\n",
    "优点：\n",
    " 由 Ingest Node 节点负责处理，职责清晰\n",
    " 更多 Processors 支持，扩展性强\n",
    " 轻量级，覆盖了 Logstash 大多常用场景\n",
    "\n",
    "PUT _ingest/pipeline/trim_pipeline\n",
    "{\n",
    "  \"processors\": [\n",
    "    {\n",
    "      \"foreach\": {\n",
    "        \"field\": \"message\",\n",
    "        \"processor\": {\n",
    "          \"trim\": {\n",
    "            \"field\": \"_ingest._value\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\n",
    "POST _ingest/pipeline/trim_pipeline/_simulate\n",
    "{\n",
    "  \"docs\": [\n",
    "    {\n",
    "      \"_source\": {\n",
    "        \"message\": [\n",
    "          \"car222 \",\n",
    "          \"auto2222 \"\n",
    "        ]\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddde36a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "Painless scripting\n",
    "\n",
    "Painless scripting 是一种简单的、安全的针对 Elasticsearch 设计的脚本语言，Painless 可以使用在任何可以使用 scripting 的场景。脚本提供了以下优点：\n",
    " 更高的性能，scripting 脚本比其他的可选脚本快数倍。\n",
    " 安全性高，更小颗粒度的字段授权机制，避免可能不必要的安全隐患安全。\n",
    " 可选类型，变量和参数可以使用显示或者动态类型编程方式。\n",
    " 语法，扩展 Java 的语法并兼容了其他脚本。\n",
    " 优化，专为 ES 设计的脚本语言。\n",
    "\n",
    "#添加测试数据\n",
    "POST my_goods/_bulk\n",
    "{\"index\":{\"_id\":1}}\n",
    "{\"goodsName\":\"苹果 51英寸 4K超高清\",\"skuCode\":\"skuCode1\",\"brandName\":\"苹果\",\"closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":8188.88,\"groupPrice\":null,\"boxPrice\":null,\"boostValue\":1.8}\n",
    "{\"index\":{\"_id\":2}}\n",
    "{\"goodsName\":\"苹果 55英寸 3K超高清\",\"skuCode\":\"skuCode2\",\"brandName\":\"苹果\",\"closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00002\",\"publicPrice\":6188.88,\"groupPrice\":null,\"boxPrice\":null,\"boostValue\":1.0}\n",
    "{\"index\":{\"_id\":3}}\n",
    "{\"goodsName\":\"苹果 UA55RU7520JXXZ 53 英寸 4K 高清\",\"skuCode\":\"skuCode3\",\"brandName\":\"美国苹果\",\"closeUserCode\":[\"0\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":8388.88,\"groupPrice\":null,\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":[\"htd003\",\"uc004\"],\"boxPriceDetail\":4388.88},{\"boxType\":\"box2\",\"boxUserCode\":[\"uc005\",\"uc0010\"],\"boxPriceDetail\":5388.88}],\"boostValue\":1.2}\n",
    "{\"index\":{\"_id\":4}}\n",
    "{\"goodsName\":\"山东苹果 UA55RU7520JXXZ 苹果 54 英寸 5K 超高清\",\"skuCode\":\"skuCode4\",\"brandName\":\"山东苹果\",\"closeUserCode\":[\"uc001\",\"uc002\",\"uc003\"],\"channelType\":\"cloudPlatform\",\"shopCode\":\"sc00001\",\"publicPrice\":8488.88,\"groupPrice\":[{\"level\":\"level1\",\"boxLevelPrice\":\"24\n",
    "88.88\"},{\"level\":\"level2\",\"boxLevelPrice\":\"3488.88\"}],\"boxPrice\":[{\"boxType\":\"box1\",\"boxUserCode\":\n",
    "[\"uc004\",\"uc005\",\"uc006\",\"uc001\"],\"boxPriceDetail\":4488.88},{\"boxType\":\"box2\",\"boxUserCode\":[\"h\n",
    "td007\",\"htd008\",\"htd009\",\"uc0010\"],\"boxPriceDetail\":5488.88}],\"boostValue\":1.2}\n",
    "\n",
    "\n",
    "# 添加字段\n",
    "POST my_goods/_update_by_query\n",
    "{\n",
    "\"script\": {\"source\":\"ctx._source.new_brandName = ctx._source.brandName + '新品'\"}}\n",
    "\n",
    "GET my_goods/_search\n",
    "\n",
    "# 删除字段\n",
    "POST my_goods/_update_by_query\n",
    "{\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.remove('new_brandName')\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# 更改字段值\n",
    "#性能较差，硬编码实现价格提升 2 倍\n",
    "POST my_goods/_update/1\n",
    "{\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.publicPrice = ctx._source.publicPrice * 2\",\n",
    "    \"lang\": \"painless\"\n",
    "  }\n",
    "}\n",
    "\n",
    "#性能较优，使用 params 将 ID 为 1 的商品的价格提高 2 倍\n",
    "POST my_goods/_update/1\n",
    "{\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.publicPrice = ctx._source.publicPrice * params.promote_percen\",\n",
    "    \"lang\": \"painless\",\n",
    "    \"params\": {\n",
    "      \"promote_percent\": 2\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#排序\n",
    "#修改 goodsName 可以被 doc 访问\n",
    "PUT my_goods/_mapping\n",
    "{\n",
    "  \"properties\": {\n",
    "    \"goodsName\": {\n",
    "      \"type\": \"text\",\n",
    "      \"fielddata\": \"true\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "#查询并排序，根据商品名称长度并添加干扰因子 1.1 倍为最终排序结果\n",
    "POST my_goods/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"match\": {\n",
    "      \"brandName\": \"苹果\"\n",
    "    }\n",
    "  },\n",
    "  \"sort\": {\n",
    "    \"_script\": {\n",
    "      \"type\": \"number\",\n",
    "      \"script\": {\n",
    "        \"lang\": \"painless\",\n",
    "        \"source\": \"doc['goodsName'].value.length() * params.factor\",\n",
    "        \"params\": {\n",
    "          \"factor\": 1.1\n",
    "        }\n",
    "      },\n",
    "      \"order\": \"asc\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "#先将脚本存储，在 DSL 查询时使用已经存储更好的脚本，叫做 stored script：\n",
    "#定义 stored script,脚本名称为：promote_price\n",
    "PUT _scripts/promote_price\n",
    "{\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.publicPrice = ctx._source.publicPrice * params.value\",\n",
    "    \"lang\": \"painless\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "POST my_goods/_update_by_query\n",
    "{\n",
    "  \"script\": {\n",
    "    \"id\": \"promote_price\",\n",
    "    \"params\": {\n",
    "      \"value\": 2\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "# source 字段访问\n",
    "# 在上面的例子中，就曾使用过 ctx._source.field_name 来更新数据\n",
    "POST my_goods/_update/1\n",
    "{\n",
    "  \"script\": {\n",
    "    \"source\": \"ctx._source.publicPrice = ctx._source.publicPrice * params.promote_percen\",\n",
    "    \"lang\": \"painless\",\n",
    "    \"params\": {\n",
    "      \"promote_percent\": 2\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 定义 pipeline\n",
    "PUT _ingest/pipeline/add_my_goods_newField\n",
    "{\n",
    "  \"processors\": [\n",
    "    {\n",
    "      \"script\": {\n",
    "        \"lang\": \"painless\",\n",
    "        \"source\": \"ctx.skuCode_brandName = ctx.skuCode + ctx.brandName\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\n",
    "#执行 pipeline\n",
    "POST my_goods/_update_by_query?pipeline=add_my_goods_newField\n",
    "{\n",
    "}\n",
    "\n",
    "\n",
    "# Painless Debug\n",
    "#定义用户信息，shop_id 为用户开的店铺 ID 信息\n",
    "PUT /user_info/_doc/1?refresh\n",
    "{\n",
    "  \"first\": \"Michael\",\n",
    "  \"last\": \"Jordan\",\n",
    "  \"shop_id\": [\n",
    "    100,\n",
    "    102,\n",
    "    103\n",
    "  ],\n",
    "  \"time\": \"2021-05-09\"\n",
    "}\n",
    "PUT /user_info/_doc/2?refresh\n",
    "{\n",
    "  \"first\": \"Michael2\",\n",
    "  \"last\": \"Jordan2\",\n",
    "  \"shop_id\": [\n",
    "    110,\n",
    "    112,\n",
    "    113,\n",
    "    114,\n",
    "    115\n",
    "  ],\n",
    "  \"time\": \"2021-05-08\"\n",
    "}\n",
    "\n",
    "#查看 mapping\n",
    "GET user_info/_mapping\n",
    "\n",
    "\n",
    "POST /user_info/_explain/1\n",
    "{\n",
    "  \"query\": {\n",
    "    \"script\": {\n",
    "      \"script\": \"Debug.explain(doc.shop_id)\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "GET user_info/_search\n",
    "{\n",
    "    \"query\": {\n",
    "        \"function_score\": {\n",
    "            \"script_score\": {\n",
    "                \"script\": {\n",
    "                    \"lang\": \"painless\",\n",
    "                    \"source\": \n",
    "                      \"\"\"\n",
    "                      return doc['shop_id'].getLength();\n",
    "                      \"\"\"\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6faaf4f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "Kibana\n",
    "\n",
    "# 查看集群支持的选项\n",
    "GET _cat\n",
    "\n",
    "# 查看节点信息\n",
    "GET _cat/nodes?v\n",
    "# 查看 Master 节点信息\n",
    "GET _cat/master?v\n",
    "\n",
    "# 查看集群状态\n",
    "#green：所有功能完好；yellow：数据是可用的，但存在未被分配的副本； red：集群中存在不可用的数据；\n",
    "GET _cat/health?v\n",
    "\n",
    "# 判断索引test是否存在\n",
    "HEAD test\n",
    "\n",
    "# 查询索引test 状态\n",
    "GET test/_stats\n",
    "\n",
    "# 打开索引\n",
    "POST test/_open\n",
    "\n",
    "# 关闭索引\n",
    "POST test/_close\n",
    "\n",
    "# 删除索引test\n",
    "DELETE test?pretty\n",
    "\n",
    "# 查看所有节点上的热点线程\n",
    "GET _nodes/hot_threads\n",
    "\n",
    "# 查看不健康的分片或索引\n",
    "GET _cluster/allocation/explain?pretty\n",
    "\n",
    "\n",
    "# 查看线程池设置\n",
    "GET _nodes/thread_pool/\n",
    "\n",
    "# 查看集群全部信息\n",
    "GET _cluster/stats?human&pretty\n",
    "\n",
    "# 查看集群状态\n",
    "GET _cluster/health?pretty\n",
    "\n",
    "\n",
    "# 获取所有索引的信息\n",
    "GET _cat/indices?v&pretty\n",
    "\n",
    "\n",
    "\n",
    "#查看所有索引摸板\n",
    "GET _template\n",
    "\n",
    "#查看以 temp 开头的索引模板\n",
    "GET _template/temp*\n",
    "\n",
    "#查看 template_1 和 template_2 索引摸板\n",
    "GET _template/template_1,template_2 \n",
    "\n",
    "#查看名称为 template_name 的索引摸板\n",
    "GET _template/template_name\n",
    "\n",
    "# 删除索引模板\n",
    "DELETE _template/template_name\n",
    "\n",
    "\n",
    "# Grok 调试\n",
    "#input\n",
    "#[2020-04-03T16:51:35,918] [DEBUG] [o.e.a.a.c.n.i.TransportNodesInfoAction] [data02-131-211]failed to execute on node [08GhVGGgRCqUE3qAdXf04g] org.elasticsearch.transport.NodeNotConnectedException: [master01-34.5][172.16.34.5:9300] Node not connected\n",
    "\n",
    "# pattern\n",
    "#(?<date>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2},\\d{3})\\] \\[(?<loglevel>[A-Z \\s]{4,5})] \\[(?<service>[A-Za-z0-9/.]{4,40})\\] \\[(?<node>[A-Za-z0-9/-]{4,40})\\] (?<msg>.*)\n",
    "\n",
    "679\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0c25cb",
   "metadata": {},
   "source": [
    "实战场景\n",
    "\n",
    "1、固定显示广告位\n",
    "PUT index_001\n",
    "{\n",
    "  \"mappings\": {\n",
    "    \"properties\": {\n",
    "      \"title\": {\n",
    "        \"type\": \"text\",\n",
    "        \"analyzer\": \"ik_max_word\",\n",
    "        \"fields\": {\n",
    "          \"keyword\": {\n",
    "            \"type\": \"keyword\"\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "PUT index_001/_bulk\n",
    "{\"index\":{\"_id\":1}}\n",
    "{\"title\":\"大众汽车首款纯电动 ID.4_预售进行时_先订先享\"}\n",
    "{\"index\":{\"_id\":2}}\n",
    "{\"title\":\"保时捷首款纯电动跑车 Taycan - 现已到店 - 电驰神往\"}\n",
    "{\"index\":{\"_id\":3}}\n",
    "{\"title\":\"纯电动电动汽车?英国国际贸易部_邀你来投资英国汽车工业\"}\n",
    "{\"index\":{\"_id\":4}}\n",
    "{\"title\":\"四轮电动车_ 电动汽车报价_阿里巴巴采购批发_超多品类低价批发\"}\n",
    "{\"index\":{\"_id\":5}}\n",
    "{\"title\":\"电动汽车之家,为新能源汽车而生 - 第一电动网\"}\n",
    "{\"index\":{\"_id\":6}}\n",
    "{\"title\":\"中国电动汽车网_新能源汽车_电动汽车网\"}\n",
    "{\"index\":{\"_id\":7}}\n",
    "{\"title\":\"电车之家_领先的电动汽车及新能源汽车行业门户网站\"}\n",
    "\n",
    "\n",
    "\n",
    "POST index_001/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"bool\": {\n",
    "      \"should\": [\n",
    "        {\n",
    "          \"match_phrase\": {\n",
    "            \"title\": {\n",
    "              \"query\": \"电动汽车\",\n",
    "              \"boost\": 5\n",
    "            }\n",
    "          }\n",
    "        },\n",
    "        {\n",
    "          \"bool\": {\n",
    "            \"should\": [\n",
    "              {\n",
    "                \"match\": {\n",
    "                  \"title\": \"电动\"\n",
    "                }\n",
    "              },\n",
    "              {\n",
    "                \"match_phrase\": {\n",
    "                  \"title\": \"汽车\"\n",
    "                }\n",
    "              }\n",
    "            ],\n",
    "            \"minimum_should_match\": 2\n",
    "          }\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "# 置顶显示id 为1、2、3的数据pinned query\n",
    "# 原理：将置顶显示的数据通过 bool 组合查询 + boost 提升权重的方式给设置了float 最大值评分，这样就能保证置顶显示了。\n",
    "GET index_001/_search\n",
    "{\n",
    "  \"query\": {\n",
    "    \"pinned\": {\n",
    "      \"ids\": [\n",
    "        \"1\",\n",
    "        \"2\",\n",
    "        \"3\"\n",
    "      ],\n",
    "      \"organic\": {\n",
    "        \"bool\": {\n",
    "          \"should\": [\n",
    "            {\n",
    "              \"match_phrase\": {\n",
    "                \"title\": {\n",
    "                  \"query\": \"电动汽车\",\n",
    "                  \"boost\": 5\n",
    "                }\n",
    "              }\n",
    "            },\n",
    "            {\n",
    "              \"bool\": {\n",
    "                \"should\": [\n",
    "                  {\n",
    "                    \"match\": {\n",
    "                      \"title\": \"电动\"\n",
    "                    }\n",
    "                  },\n",
    "                  {\n",
    "                    \"match_phrase\": {\n",
    "                      \"title\": \"汽车\"\n",
    "                    }\n",
    "                  }\n",
    "                ],\n",
    "                \"minimum_should_match\": 2\n",
    "              }\n",
    "            }\n",
    "          ]\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "2、ELK 日志采集处理\n",
    "策略：按天产生 index，一些 IP，APP 应用名等不需要分词查询的字段都禁用了 index(这样可以节省磁盘)，只保留一周的日志回溯，3 天的日志 alive 查询，4 天的日志close。一周以上的 index 直接 delete ，晚上 12 点 定时执行 forcemerge。\n",
    "\n",
    "\n",
    "集群健康程度可以从几个方面表现：cluster 层面、node 层面、shard 层面、index 层面、jvm 层面、threadpool 层面。如下为参考指标："
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myenv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
